What Is the Cluster Sweep Node?

One way to parallelize the computation of a parameter set is to combine the parametric sweep with the Cluster Computing node. When doing so, you create one large cluster job that spans a number of nodes. The more nodes you add, the more parameter values are computed in parallel (as long as there are more parameters than nodes, of course).

A cluster example.

You can also use the Cluster Sweep node to parallelize computations. It is designed for when you want to split up a parametric sweep into several cluster computing jobs. You define a list of parametric values in the Cluster Sweep node. For each of these values, a separate batch job is sent to your cluster queue. When the computations are done, COMSOL Multiphysics incorporates the results back into the main process.

You can even nest parametric sweeps this way, combining the cluster sweep with a “normal” parametric sweep. You decide which parameters you start separate jobs for and which parameters you want to keep “inside” the jobs.

In short, the Cluster Sweep node is a powerful tool that COMSOL Multiphysics supplies you with to help you be in full control of your modeling process.

Note that to use a cluster sweep, a Floating Network License (FNL) is required. It is also recommended that you are familiar with the settings discussed in this blog post: How to Run on Clusters from the COMSOL Desktop® Environment. If you follow the steps in that blog post and save your settings, they will automatically be used in the Cluster Sweep node.

When to Use the Cluster Sweep Node

By now, you know what the Cluster Sweep node is, and you might find yourself wondering two things:

1. When should I use it?
2. When is it preferable over the Cluster Computing node?

The first case that comes to mind is when you have a parameter set and you don’t know if your model will converge or even be valid for all parameter combinations. Your parameter set could control your geometry and, for some values, the geometry causes your solving or meshing to fail. If you compute the model with a parametric sweep, COMSOL Multiphysics cancels the computation at the first failing geometry — even if later ones would finish. By splitting this computation into individual jobs, your computations will be started for each parameter value.

Another situation is when the amount of interesting parameter values is simply too large to be feasible for a single cluster job. If you have parameters that control the frequency, geometry, materials, boundary conditions, and so on, you end up with a lot of computations if you want results for all available combinations. If you put all of these computations into one large job and send it to your cluster, you will almost surely end up with an unhappy cluster administrator and a lot of angry colleagues (more on that later).

Good news: Using the Cluster Sweep node, you can split your potentially enormous job into several smaller ones. To do so, you add a Parametric Sweep node to your model in addition to the Cluster Sweep node. Setting up your model in this way creates what is known as a nested parametric sweep (similar to a nested for-loop in programming). To learn how to do this, keep reading. We’ve included a short tutorial in this blog post.

There is one more thing to note about the Cluster Sweep node: You can use it to potentially increase the throughput of jobs on your cluster.

Using a Cluster Sweep to Optimize Scheduling

Earlier in this blog post, I mentioned an unhappy cluster admin, and you might be wondering why. Computation time is a valuable resource on high-performance clusters. Because of this, most clusters have some kind of queue or scheduling system implemented. How large jobs are handled is up to the cluster administrator, and a rule of thumb is that large jobs mean long waiting times. Why? Large jobs occupy a lot of computational resources, and they can take a long time to complete. Hence, in order to not hold up other users’ jobs, large jobs are assigned a low priority. Of course, this all depends on how your cluster admin has configured the scheduler; that is, your mileage may vary.

What does this have to do with the Cluster Sweep node? Suppose that you have access to a cluster where it is hard to get a large job scheduled, but smaller ones are easier, since they fill the gaps in the scheduler (an unused cluster node is an expensive cluster node). You can use a cluster sweep to split the large job into small ones.

Let’s look at it with the help of an example: Instead of starting 1 large job on 8 nodes to parallelize 800 parameter values, you can start 8 jobs that each use 1 node to compute their own sets of 100 parameter values. The jobs will then be scheduled independently and, depending on how your cluster is set up, the small jobs might finish faster than the large job would!

Setting Up Cluster Sweeps and Nested Parametric Sweeps

If you have previous experience with the batch sweep and the Cluster Computing node, using the Cluster Sweep node is easy. (To find descriptions on how to set up a batch sweep, check out the blog posts “The Power of the Batch Sweep” and “Added Value of Task Parallelism in Batch Sweeps“.)

To demonstrate how to set up both a pure cluster sweep and a nested parametric sweep, let’s turn to my favorite example model: the parameterized thermal microactuator. (It’s my favorite because the model shows the multiphysics capabilities of the COMSOL® software.) Since it is parameterized, it’s very easy to add parametric and cluster sweeps to the model.

Modeling the Joule heating of a microactuator. Current flows through two of the arms, causing them to heat up. Then, the thermal expansion causes the actuator to bend.

Adding a Cluster Sweep

We start by adding a cluster sweep over the actuator length parameter, called L. To do so, first right-click Study 1 and click Cluster Sweep. This adds a node where you can set up your cluster settings, analogue to the instructions in this blog post on running clusters from the COMSOL Desktop®. (If you haven’t read the blog post yet, now is a good time.)

Next, in the Study Settings window, you can add the parameters that you want to sweep over. Click the plus symbol and, in the drop-down list, choose the parameter L. Then, in the Parameter value list field, write (for example) “100 170 240 310”. In the Parameter unit field, write “um” (for micrometers).

If you want to bring the results back into your main model, make sure to check the Synchronize solutions check box. This way, you’ll have all the results available for further analysis and postprocessing.

Adding a cluster sweep to the model.

We have now created a cluster sweep, which loops over the length of the microactuator. Assuming that the cluster settings are correct, all we need to do now is click Compute and the separate jobs are sent to the cluster.

Adding a Nested Parametric Sweep

Now, let’s create a nested parametric sweep so that each of our cluster jobs contains a parametric sweep themselves. We do this by adding a parametric sweep over the voltage parameter, called DV. To do so, follow these steps:

1. Right-click Study 1 and click Parametric Sweep, which adds a node where you can set up a parametric sweep
2. In the Study Settings window, click the plus symbol and, in the drop-down list, choose the parameter DV
3. In the Parameter value list field, write “1 2 3 4 5”
4. In the Parameter Unit field, write “V”
5. Click Compute so that COMSOL Multiphysics will schedule the jobs for you

Adding a parametric sweep to the model, which creates a nested parametric sweep.

You can either wait for the jobs to complete (their status is shown in the External Processes window) or you can detach from the processes, save the model, close COMSOL Multiphysics, and let the jobs run themselves. When you come back to your workstation, just open your saved model and reattach, and the software will process the results just as when using the regular Cluster Computing node. This workflow is perfect for overnight simulations!

Concluding Thoughts

In this blog post, you have learned how you can optimize the parallelization of parametric computations on clusters using the Cluster Sweep node. You have also learned when it is beneficial to use different approaches and, as a bonus, how to avoid making your system admin unhappy.

As with cluster computing in general, you must decide what approach to use depending on the model you want to compute. To know when to use a cluster sweep and a distributed parametric sweep, you have to try the two approaches on your models and your cluster. As always, to master something, you have to test it!

As mentioned, you’ll need an FNL to use a cluster sweep, since this functionality is a network-based technology.

Next Step

If you want to learn more about the Cluster Sweep node, you can contact us by simply clicking the button below.

Linux is a registered trademark of Linus Torvalds in the U.S. and other countries.

What Is Cluster Computing?

The most common type of HPC hardware is a cluster; a bunch of individual computers (often called nodes) connected by a network. Even if there is only one dedicated simulation machine, you can think of it as a one-node cluster.

The COMSOL Reference Manual also calls a single COMSOL Multiphysics process a node. The difference is rarely important, but when it does matter, we will call a computer a physical node or host and an instance of the COMSOL Multiphysics program a compute node or process.

An example of a cluster with four compute nodes.

The work that we want to perform on the cluster is bundled into atomic units, called jobs, that are submitted to the cluster. A job in this context is a study being run with COMSOL Multiphysics.

When you submit a job to a cluster, the cluster does two things:

• Decides which nodes run which jobs and at what time
• Restricts the access to the nodes, so multiple jobs do not interfere with each other

These tasks are performed by special programs called schedulers and resource managers, respectively. Here, we use the term scheduler interchangeably for both terms, since most programs perform both tasks at once, anyway.

Note that it is possible to submit COMSOL Multiphysics jobs to a cluster using the comsol batch command (on the Linux® operating system) or comsolbatch.exe (on the Windows® operating system) in a script that you submit to the cluster. You might prefer this method if you’re already familiar with console-based access to your cluster. For additional information, please see the COMSOL Knowledge Base article “Running COMSOL® in parallel on clusters“.

In the following sections, we will discuss using the Cluster Computing node to submit and monitor cluster jobs from the COMSOL Desktop® graphical interface.

Adding the Cluster Computing Node to a Simple Model

Whenever I want to configure the Cluster Computing node for a cluster that I am not familiar with yet, I like to start with a simple busbar model. This model solves in a few seconds and is available with any license, which makes testing the cluster computing functionality very easy.

To run the busbar model on a cluster, we add the Cluster Computing node to the main study. We might need to enable Advanced Study Options first, though. To do so, we activate the option in Preferences or click the Show button in the Model Builder toolbar.

Activate Advanced Study Options to enable the Cluster Computing node.

Now the Cluster Computing node can be added to any study by right-clicking the study and selecting Cluster Computing.

Right-click a study node and select Cluster Computing from the menu to add it to the model.

The default settings for the Cluster Computing node.

If you can’t find the Cluster Computing node, chances are your license is not cluster enabled (such as CPU licenses and academic class kit licenses). In this case, you can contact your sales representative to discuss licensing options.

Settings for the Cluster Computing Node

The most complex part of using the Cluster Computing node is finding the right settings and using it for the first time. Once the node works on your cluster for one model, it is very straightforward to adjust the settings slightly for other simulations.

To store the settings as defaults, you can change the settings under Preferences in the sections Multicore and Cluster Computing and Remote Computing. Alternatively, you can apply the default settings to the Cluster Computing node directly and click the Save icon at the top of the Settings window. It is highly recommended to store the settings as default settings either way, so you do not have to type everything again for the next model.

Discussing all of the possible settings for the Cluster Computing node is out of scope of this blog post, so we will focus on a typical setup. The COMSOL Multiphysics Reference Manual contains additional information. In this blog post, the following is assumed:

• COMSOL Multiphysics® is running on a local Windows® machine and we want to submit jobs to a remote cluster
• The cluster is running on Linux® and has SLURM® software installed as the scheduler

These settings are shown in this screenshot:

First, let’s talk about the section labeled Cluster computing settings. Since our cluster uses SLURM® software as its scheduler, we set the Scheduler type to “SLURM”. The following options are SLURM®-specific:

• Scheduler is left empty to instruct SLURM® software to just use the one scheduler that is available
• User is our username, which can be left empty to use the username we log in with on the cluster
• Queue name is the name of the queue to which the job is submitted

On the machine used in this example, we have two queues: “cluster” for jobs of up to 10 physical compute nodes with 64 GB of RAM each and “fatnode” for a single node with 256 GB. Every cluster will have different queues, so ask your cluster administrator what queues to use.

The next field is labeled “Directory”. This is where the solved COMSOL Multiphysics files go on a local computer when the job is finished. This is also where the COMSOL® software will store any intermediate, status, and log files.

The next three fields specify locations on the cluster. Notice that Directory was a Windows® path (since we are working on a Windows® computer here), but these are Linux® paths (since our cluster uses Linux®). Make sure that the kind of path matches the operating system on the local and remote side!

The Server Directory specifies where files should be stored when using cluster computing from a COMSOL Multiphysics session in client-server mode. When executing cluster computing from a local machine, this setting is not used, so we leave it blank. We do need the external COMSOL batch directory, however. This is where model files, status files, and log files should be kept on the cluster during the simulation. For these paths, be sure to choose a directory that already exists where you have write permissions; for example, some place in your home directory. (See this previous blog post on using client-server mode for more details.)

The COMSOL installation directory is self-explanatory and should contain the folders bin, applications, and so on. This is usually something like “/usr/local/comsol/v53a/multiphysics/” by default, but it obviously depends on where COMSOL Multiphysics is installed on the cluster.

Remote connection settings.

The next important section is the Remote and Cloud Access tab. This is where we specify how to establish the connection between the local computer and remote cluster.

To connect from a Windows® workstation to a Linux® cluster, we need the third-party program PuTTY to act as the SSH client for the COMSOL® software. Make sure to have PuTTY installed and that you can connect to your cluster with it. Also, make sure that you set up password-free authentication with a public-private key pair. There are many tutorials online on how to do this and your cluster administrator can help you. When this is done, enter the installation directory of PuTTY as the SSH directory and your private key file from the password-free authentication in the SSH key file. Set the SSH user to your login name on the cluster.

While SSH is used to log in to the cluster and run commands, SCP is used for file transfer, for example, when transferring model files to or from the cluster. PuTTY uses the same settings for SCP and SSH, so just copy the settings from SSH.

Lastly, enter the address of the cluster under Remote hosts. This may be a host name or an IP address. Remember to also set the Remote OS to the correct operating system on the cluster.

When you are done, you can click the Save icon at the top of the Settings window to start with these settings next time you want to run a remote cluster job.

Another possibility to test whether your cluster settings work is to use the Cluster Setup Validation app, available as of COMSOL Multiphysics version 5.3a.

Running a Study on a Cluster

The settings that change every time you run a study include the model name and the number of physical nodes to use. When you click to run the study, COMSOL Multiphysics begins the process of submitting the job to the cluster. The first step is invisible and involves running SCP to copy the model file to the cluster. The second step is starting the simulation by submitting a job to the scheduler. Once this stage starts, the External Process window automatically appears and informs you of the progress of your simulation on the cluster. During this stage, the COMSOL Desktop® is locked and the software is busy tracking the remote job.

Tracking the progress of the remote job in the External Process window from scheduling the job (top) to Done (bottom).

This process is very similar to how the Batch Sweep node works. In fact, you may recognize the External Process window from using the batch sweep functionality. Just like when using a batch sweep, we can regain control of the GUI by clicking the Detach Job button below the External Process window, to detach the GUI from the remote job. We can later reattach to the same job by clicking the Attach job button, which replaces the Detach job button when we are detached.

Normally, running COMSOL Multiphysics on two machines simultaneously requires two license seats, but you can check the Use batch license option to detach from a remote job and keep editing locally with only one license seat. In fact, you can even submit multiple jobs to the cluster and run them simultaneously, as long as both jobs are just variations of the same model file; i.e., they only differ in their global parameter values. The only restriction is that your local username needs to be identical to the username on the remote cluster so the license manager can tell that the same person is using both licenses. Otherwise, an extra license seat will be consumed, even when the Use batch license option is enabled.

As soon as the simulation is done, you are prompted to open the resulting file:

Once the cluster job has finished, you are prompted to immediately open the solved file.

If you select No, you can still open the file later, because it will have already been downloaded and copied to the directory that was specified in the settings. Let’s have a look at these files:

Files created during the cluster job on the local side.

These files are created and updated as the simulation progresses. COMSOL Multiphysics periodically retrieves each file from the remote cluster to update the status in the Progress window and informs you as soon as the simulation is done. The same files are also present on the remote side:

Files created during the cluster job on the remote side. Note: Colors have been changed from the default color scheme in PuTTY to emphasize MPH-files.

Here is a rundown of the most relevant file types:

Using Cluster Computing Functionality for Your COMSOL Multiphysics® Simulations

The busbar model, being so small, is not something that we would want to realistically run on a cluster. After using that example to test the functionality, we can open up any model file, add the Cluster Computing node (populated with the defaults we set before), change the number of nodes and filename, and click Compute. The Run remote options, scheduler type, and all of the associated settings don’t need to be changed again.

What does the COMSOL® software do when we run a model on multiple hosts? How is the work split up? Most algorithms in the software are parallelized, meaning the COMSOL Multiphysics processes on all hosts work together on the same computation. Distributing the work over multiple computers provides more computing resources and can increase performance for many problems.

However, it should be noted that the required communication between cluster nodes can produce a performance bottleneck. How fast the model will solve depends a lot on the model itself, the solver configuration, the quality of the network, and many other factors. You can find more information in this blog series on hybrid modeling.

Another reason to use the hardware power of a cluster is that the total memory that a simulation needs stays approximately constant, but there is more memory among all of the hosts, so the memory needed per host goes down. This allows us to run really large models that would not otherwise be possible to solve on a single computer. In practice, the total memory consumption of the problem goes up slightly, since the COMSOL Multiphysics processes need to track their own data as well as the data they receive from each other (usually much less). Also, the exact amount of memory a process will need is often not predictable, so adding more processes can increase the risk that a single physical node will run out of memory and abort the simulation.

A much easier case is running a distributed parametric sweep. We can speed up the computation by using multiple COMSOL Multiphysics processes and having each work on a different parameter value. We call this type of problem “embarrassingly parallel”, since the nodes do not need to exchange information across the network at all while solving. In this case, if the number of physical nodes is doubled, then ideally the simulation time will be cut in half. The actual speedup is typically not quite this good, as it takes some time to send the model to each node and additional time to copy the results back.

To run a distributed parametric sweep, we need to activate the Distribute parametric sweep option at the bottom of the settings for the parametric sweep. Otherwise, the simulation will run one parameter at a time using all of the cluster nodes, with the parallelization performed on the level of the solver, which is much less efficient.

If you run an auxiliary sweep, you can also check the Distribute parametric solver option in the study step, for example, to run a frequency sweep over many frequencies in parallel using multiple processes on potentially many physical nodes. Note that if you use a continuation method, or if individual simulations depend on each other, then this method of distributing the parameters does not work.

Note: Do not use the Distribute parametric sweep option in the Cluster Computing node itself, as it has been depreciated. It is better to specify this at the parametric sweep directly.

Activate the Distribute parametric sweep option to run each set of parameters on a different node in parallel.

To run a sweep in parallel, we can also use the Cluster Sweep node, which combines the features of the Batch Sweep node with the ability of the Cluster Computing node to run jobs remotely. You can say that a cluster sweep is the remote version of the batch sweep, just like the Cluster Computing node is the remote version of the Batch node. We will discuss cluster sweeps in more detail in a future blog post.

The most important difference to remember is that the Cluster Computing node submits one job for the entire study (even if it contains a sweep), while the Cluster Sweep and Batch Sweep nodes create one job for each set of parameter values.

Cluster Computing with Apps

All of what is covered in this blog post is also available from simulation apps that are run from either COMSOL Multiphysics or COMSOL Server™. An app simply inherits the cluster settings from the model on which it is based.

When running apps from COMSOL Server™, you get access to cluster preferences in the administration web page of COMSOL Server™. You can let your app use these preferences to have the cluster settings hardwired and customized for a particular app. If you wish, you can design your apps so that the user of the app gets access to one or more of the low-level cluster settings. For example, in your app’s user interface, you can design a menu or list where users can select between different queues, such as the “cluster” or “fatnode” options mentioned earlier.

Concluding Thoughts

Whether you are using a university cluster, a virtual cloud environment, or your own hardware, the Cluster Computing node enables you to easily run your simulations remotely. You don’t usually need an expensive setup for this purpose. In fact, sometimes all you need is a Beowulf cluster for running parametric sweeps while you take care of other tasks locally.

Cluster computing is a powerful tool to speed up your simulations, study detailed and realistic devices, and ultimately help you with your research and development goals.

SLURM is a registered trademark of SchedMD LLC.

Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

Three Cases Where You Need More Computer Power

Imagine that you are modeling an electronic device and are interested in its temperature distribution during operation. After testing a few setups, you discover that the heat flux boundary condition you applied is not a well-suited approximation for your model. You realize that a fluid flow simulation is required in order to achieve more accurate results. The only problem is that you have used almost all of your laptop’s 4 GB of RAM to model the heat transfer simulation, based on the heat flux approximation. You require two-way coupling, and including the simulation of fluid flow will only add even more degrees of freedom to your computation — and require even more RAM.

What now? You need more computer power.

Now imagine, instead, that you are analyzing the mechanics of a structural component with a lot of small details for your customer. In order to optimize the design, you are required to run the analysis for a large number of different design dimensions. As you have only the one processor locally and each run will take quite a bit of time, you realize that you will not reach your customer’s deadline.

The solution? You would need to run these simulations in parallel on multiple processors.

Finally, let’s look at an application independent of the physics involved, but still reliant on the analysis performed. You have set up your model using the physics interface of your choice, but it’s the end of the day and you just want to get the computed solution to your model as quickly and easily as possible, overnight. Utilizing a direct solver does not require a lot of work with manipulating solver settings and the like, but the required RAM of a direct solver increases greatly with the number of degrees of freedom in your model.

What’s the fix this time? You need a bigger computer.

What if there was another solution to all three cases…

Enter: Cloud Computing

This is where cloud computing comes into the picture. Compute clouds are services that make computing power available to those who need it, when they need it.

The service has several advantages, especially if you don’t have the time, money, and experience to invest in a traditional cluster or server rack. You might also not need a cluster available 24/7, but only need that extra compute power during certain periods of time, for instance, for that one-off analysis or task that needs to be performed quicker.

An organization can access COMSOL Multiphysics® and the hardware resources of cloud computing to run many different analyses at the time they require, utilizing the resources they require.

Utilizing cloud computing will have a positive impact on your workflow. The ability to add more compute power directly when you need it will enable you to be more agile in your day-to-day COMSOL Multiphysics® simulation work. You won’t have to worry about the lack of adequate hardware on-site and you can go about your daily business with the certainty that you can expand into the cloud whenever the situation calls for it.

Using the COMSOL® Software on Remote Computing Resources

Traditionally, when using cloud computing services, you need expertise in the network and hardware technology being used, as well as the operating system and software implemented by the cloud service to support running your application. In an example workflow, you register to the cloud service, research what specifications their machines require, rent the machine, and then connect it to your network to allow access to your license server. Next comes the easy part: installing COMSOL Multiphysics® and running your model.

However, since HPC is becoming more and more important in the CAE community, we have partnered with cloud computing providers to make it as simple as possible for you to take the step into the cloud.

Note: COMSOL Multiphysics has been able to utilize remote computing resources for a long time, either through batch jobs started from the user interface or the command line, or on-the-fly through client-server technology. For this, you only need a Floating Network License (FNL) for COMSOL Multiphysics®.

Get Started: Contact Our Cloud Computing Partners

• Nimbix
• Rescale
• CPU 24/7
• Nor-Tech

Let’s Look at Some Data

The plot below shows the amount of memory needed to solve various different 3D finite element problems in terms of the number of degrees of freedom (DOF) in the model.

Memory requirements (with a second-polynomial curve fit) with respect to degrees of freedom for various representative cases.

There are five different cases presented here:

• Case 1: A heat transfer problem of a spherical shell. There is radiative heat transfer between all of the surfaces. The model is solved with the default iterative solver.
• Case 2: A structural mechanics problem of a cantilevered beam, solved with the default direct solver.
• Case 3: A wave electromagnetics problem solved with the default iterative solver.
• Case 4: The same structural mechanics problem as Case 2, but using an iterative solver.
• Case 5: A heat transfer problem of a block of material. Only conductive heat transfer is considered. The model is solved with the default iterative solver.

What you should see from this graph is that, with a computer that has 64 GB of random access memory (RAM), you can solve problems that range in size anywhere from ~26,000 DOF on the low end all the way up to almost 14 million degrees of freedom. So why this wide range of numbers? Let’s look at how to interpret these data…

Degrees of Freedom, Explained

For most problems, COMSOL Multiphysics solves a set of governing partial differential equations via the finite element method, which takes your CAD model and subdivides the domains into elements, which are defined by a set of nodes on the boundaries.

At each node, there will be at least one unknown, and the number of these unknowns is based upon the physics that you are solving. For example, when solving for temperature, you only have a single unknown (called T, by default) at each node. When solving a structural problem, you are instead computing strains and the resultant stresses, thus you are solving for three unknowns (u,v,w), which are the displacements of each node in the x-y-z space.

For a turbulent fluid flow problem, you are solving for the fluid velocities (also called u,v,w by default) and pressure (p) as well as extra unknowns describing the turbulence. If you are solving a diffusion problem with many different species, you will have as many unknowns per node as you have chemical species. Additionally, different physics within the same model can have a different default discretization order, meaning there can be additional nodes along the element edges, as well as in the element interior.

A second-order tetrahedral element solving for the temperature field, , will have a total of 10 unknowns per element, while a first-order element solving the laminar Navier-Stokes equations for velocity, , and pressure, , will have a total of 16 unknowns per element.

COMSOL Multiphysics will use the information about the physics, material properties, boundary condition, element type, and element shape to assemble a system of equations (a square matrix), which need to be solved to get the answer to the finite element problem. The size of this matrix is the number of degrees of freedom (DOFs) of the model, where the number of DOFs is a function of the number of elements, the discretization order used in each physics, and the number of variables solved for.

These systems of equations are typically sparse, which means that most of the terms in the matrix are zero. For most types of finite element models, each node is only connected to the neighboring nodes in the mesh. Note that element shape matters; a mesh composed of tetrahedra will have different matrix sparsity from a mesh composed of hexahedra (brick) elements.

Some models will include non-local couplings between nodes, resulting in a relatively dense system matrix. Radiative heat transfer is a typical problem that will have a dense system matrix. There is radiative heat exchange between any surfaces than can see each other, so each node on the radiating surfaces is connected to every other node. The result of this is clearly seen in the plots I shared at the beginning of this blog post. The thermal model that includes radiation has much higher memory requirements than the thermal model without radiation.

You should see, at this point, that it is not just the number of DOFs, but also the sparsity of the system matrix that will affect the amount of memory needed to solve your COMSOL Multiphysics model. Let’s now take a look at how your computer manages memory.

How Your Operating System Manages Memory

COMSOL Multiphysics uses the memory management algorithms provided by the Operating System (OS) that you are working with. Regardless of which OS you are using, the performance of these algorithms is quite similar on all of the latest OS’s that we support.

The OS creates a Virtual Memory Stack, which the COMSOL software sees as a continuous space of free memory. This continuous block of virtual memory can actually map to different physical locations, so some part of the data may be stored within RAM and other parts will be stored on the hard disk. The OS manages where (in RAM or on disk) that the data is actually stored, and by default you do not have any control over this. The amount of virtual memory is controlled by the OS, and it is not something that you usually want to change.

Under ideal circumstances, the data that COMSOL Multiphysics needs to store will fit entirely within RAM, but once there is no longer enough space, part of the data will spill over to the hard disk. When this happens, performance of all programs running on the computer will be noticeably degraded.

If too much memory space is requested by the COMSOL software, then the OS will determine that it can no longer manage memory efficiently (even via the hard disk) and will tell COMSOL Multiphysics that there is no more memory available. This is the point at which you will get an out-of-memory message and COMSOL Multiphysics will stop trying to solve the model.

Next, let’s take a look at what COMSOL Multiphysics is doing when you get this out-of-memory message and what you can do about it.

When Does COMSOL Use the Most Memory?

When you set up and solve a finite element problem, there are three memory intensive steps: Meshing, Assembly, and Solving.

• Meshing: During the meshing step, the CAD geometry is subdivided into finite elements. The default meshing algorithm applies a free tetrahedral mesh over most of the modeling space. Free tetrahedral meshing of large complex structures will require a lot of memory. In fact, it can sometimes require more memory than actually solving the system of equations, so it is possible to run out of memory even at this step. If you do find that meshing is taking significant time and memory, then you should subdivide (or partition) your geometry into smaller sub-domains. Generally, the smaller the domains, the less memory intensive they are to mesh. By meshing in a sequence of operations, rather than all at once, you can reduce the memory requirements. Within the context of this blog entry, it is also assumed that there are no modeling simplifications (such as exploiting symmetry or using thin layer boundary conditions) that could be leveraged to simplify the model and reduce the mesh size.
• Assembly: During the assembly step, COMSOL Multiphysics forms the system matrix as well as a vector describing the loads. Assembling and storing this matrix requires significant memory — possibly more than the meshing step — but always less than the solution step. If you run out of available memory here, you should increase the amount of RAM in your system.
• Solving: During the solution step, COMSOL Multiphysics employs very general and robust algorithms capable of solving nonlinear problems, which can consist of arbitrarily coupled physics. At the very core of these algorithms, however, the software will always be solving a system of linear equations, and this can be done using either direct or iterative methods. So let’s look at these two methods from the point of view of when they should be used and how much memory they need.

Direct Solvers

Direct solvers are very robust and can handle essentially any problem that will arise during finite element modeling. The sparse matrix direct solvers used by COMSOL Multiphysics are the MUMPS, PARDISO, and SPOOLES solvers. There is also a dense matrix solver, which should only be used if you know the system matrix is fully populated.

The drawback to all of these solvers is that the memory and time required goes up very rapidly as the number of DOFs and the matrix density increase; the scaling is very close to quadratic with respect to number of DOFs.

As of writing this, both the MUMPS and PARDISO direct solvers in the COMSOL software come with an out-of-core option. This option overrides the OS’s memory management and lets COMSOL Multiphysics directly control how much data will be stored in RAM and when and how to start writing data to the hard drive. Although this is superior to the OS’s memory management algorithm, it will be slower than solving the problem entirely in RAM.

If you have access to a cluster supercomputer, such as the Amazon Web Service™ Amazon Elastic Compute Cloud™, you can also use the MUMPS solver to distribute the problem over many nodes of the cluster. Although this does allow you to solve much larger problems, it is also important to realize that solving on a cluster may be slower than solving on a single machine.

Due to their aggressive (approximately quadratic) scaling with problem size, the direct solvers are only used as the default for a few of the 3D physics interfaces (although they are almost always used for 2D models, for which their scaling is much better.)

The most common case where the direct solver is used by default is for 3D structural mechanics problems. While this choice has been made for robustness, it is also possible to use an iterative solver for many structural mechanics problems. The method for switching the solver settings is demonstrated in the example model of the stresses in a wrench.

Iterative Solvers

Iterative solvers require relatively much less memory than the direct solvers, but they require more customization of the settings to get them to work well.

With all of the predefined physics interfaces where it is reasonable to do so, we have provided default iterative solver suggestions that are selected for robustness. These settings are handled automatically and do not require any user interaction, so as long as you are using the built-in physics interfaces, you do not need worry about these settings.

The memory and time needed by an iterative solver will be relatively much less than a direct solver for the same problem, so when they can be used, they should be. The scaling as the problem size increases is much closer to linear, as opposed to the quadratic scaling typical of the direct solvers.

At the time of writing this, the iterative solvers should be used on a computer that has enough RAM to solve the problem, so if you get an out-of-memory message when using an iterative solver, you should upgrade the amount of RAM on your computer.

It is also possible to use an iterative solver on a cluster computer using Domain Decomposition methods. This class of iterative methods has recently been introduced into the software, so stay tuned for more details about this in the future.

Predicting Memory Requirements

Although the data shown above do provide an upper and lower bound of memory requirements, these bounds are quite wide. We’ve seen that introducing a small change to a model, such as introducing a non-local coupling like radiative heat transfer, can significantly change memory requirements. So let’s introduce a general recipe for how you can predict memory requirements.

Start with a representative model that contains the combination of physics you want to solve and approximates the true geometric complexity. Begin with as coarse a mesh as possible, and then gradually increase the mesh refinement. Alternatively, start with a smaller representative model and gradually increase the size.

Solve each model and monitor memory requirements. Observe the default solver being used. If it is a direct solver, use the out-of-core option in your tests, or consider if an iterative solver can be used instead. Fit a second-order polynomial to the data, and use this curve to predict the memory required by the size of the larger problem that you eventually want to solve. This is the most reliable way to predict the memory requirements of large, complex, 3D multiphysics models.

As we have now seen, the memory needed will depend upon (at least) the geometry, mesh, element types, combination of physics being solved, couplings between the physics, and the scope of any non-local model couplings. At this point, it should also be made clear that it is not generally possible to predict the memory requirements in all cases. You may need to repeat this procedure several times for variations of your model.

It is also fair to say that setting up and solving large models in the most efficient way possible is something that can require some deep expertise of not just the solver settings, but also of finite element modeling in general. If you do have a particular modeling concern, please contact your COMSOL Support Team for guidance.

Summary

You should now have an understanding of why the memory requirements for a COMSOL Multiphysics model can vary dramatically. You should also be able to predict with confidence the memory requirements of your larger models and decide what kind of hardware is appropriate for your modeling challenges.

Amazon Web Services and Amazon Elastic Compute Cloud are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.

A Very Simple Model of a Typical Desktop Computer

Let’s start by considering a very simplified model of a computer, composed of just three parts: random access memory (RAM), which is used to store information; a processing unit, which performs mathematical operations on the information; and a bus, which transfers the data between the two.

Schematic of the key parts of a computer.

For the purposes of this blog post, let’s imagine that all of the data about the problem is sitting in the RAM and that this data gets moved over to the processing unit via the bus. The memory bus itself can be composed of several channels operating in parallel, effectively increasing throughput. The processing unit can be composed of several chips, each of which can have several computational cores that are able to work on data simultaneously, after it has been loaded from the memory via the bus. Let’s use this as our mental model of the computer sitting on our desktop.

Shall We Play a Game?

Many problems in computer science can be thought of as games that we played as children. Let’s look at three of the classics.

Finding Walter (Waldo)

First, let’s try to find a face in the crowd, á la Where’s Waldo?

A photo from the COMSOL Conference. Can you find me?

Suppose we have a photo with hundreds of people — what’s the fastest way of finding one person?

You could scan through the entire image by yourself, checking faces one by one to see if they match the person you are searching for. But, this can be quite slow. You can also invite your friends over to help. In this case, you would first subdivide the picture into smaller pieces. Each person can then independently work on one piece at a time.

In the language of computer science, we would say that this game is completely parallel.

Having two people working will halve the solution time, four people will cut the solution time in four, and so on. But, there is a limit — you can only have as many friends helping you as there are faces in the crowd. Beyond that point, inviting more people to help won’t speed up the process, and it may even slow things down.

Solving a Puzzle

Next, let’s try to solve a jigsaw puzzle.

Can you put together the image?

This is a bit more complicated — you can have multiple people working at once, but they cannot work independently. Each person will take a few dozen puzzle pieces for themselves from the main pile and try to fit them together with the pieces that their friends are putting together. People will try to fit their own pieces together. They will pass pieces back and forth, and they will be in constant communication with each other.

A computer scientist would call this a partially parallel game.

Although adding more people will decrease the solution time, it will not be a simple mathematical relationship. Suppose you have a 1,000-piece puzzle, and 10 people with 100 pieces each. They will spend relatively more time working on their own pieces and less time talking. On the other hand, if you have 100 people with 10 pieces each, there will be a lot more talking and moving pieces around. And what will happen when you have 1,000 people working on a puzzle with 1,000 pieces? Try that one at home!

You can probably see that for a puzzle of a certain size, there is some maximum number of people that can be working it. This number will be much lower than the number of puzzle pieces. Adding more people won’t speed things up noticeably.

Stacking Blocks

Finally, let’s try to stack some blocks on top of each other to form a tower, and then raise the height of the tower by taking the blocks from the lower levels and adding them to the top without causing the structure to topple over.

How high can you stack the tower? (JENGA® tower standing on one tile. “Jenga distorted” by Guma89 — Own work. Licensed under Creative Commons Attribution-Share Alike 3.0 via Wikimedia Commons.)

In this game, only one person can play at a time, and we can say that the game is completely serial.

Playing with more people won’t finish the game any faster, and if you invite too many people to play, some of them will never get a chance to do anything. In fact, it’s probably fastest (albeit not very sociable) to play this game by yourself.

How Does This Relate to COMSOL Multiphysics®?

You can probably already see the relationships between playing these games and using COMSOL Multiphysics. How about we start classifying the problems you solve in COMSOL Multiphysics into these categories:

• Partially Parallel — All problems in COMSOL Multiphysics have a partially parallel component. Solving a system of linear equations is a partially parallelizable problem. Thus, no matter what class of problem you are solving, some (usually significant) fraction of the solution time is spent solving a partially parallel problem. For problems where you are solving a stationary, frequency-domain, or eigenfrequency problem only, almost all of the time is spent solving the system of linear equations.
• Completely Parallel — A completely parallel problem arises when you use the Parametric Sweep functionality, such as when sweeping over a range of geometric dimensions. Each step in the parameter sweep will need to solve a partially parallelizable problem, but each parameter can be solved independently, and if you solve the parameters independently there is no information exchanged between the various cases.
• Completely Serial — A serial problem arises when subsequent parts of the solution depend on previously computed values. Time-dependent models, models using continuation methods, and optimization problems fall into this category. All such models still need to solve a system of linear equations, but they do so sequentially. The possible speedup is mainly governed by the speedup possible when solving the partially parallel system of linear equations.

How Does This Relate to My Desktop Hardware?

When solving, COMSOL Multiphysics is spending most of its time solving a partially parallel problem. So, what actually happens in the hardware?

The COMSOL software starts with the information about the loads, boundary conditions, material properties, and finite element mesh and generates data that is used during the solution process. Many gigabytes of memory are needed to store the system matrices, generate intermediate information, and compute the solution. Ideally, all of this information should be stored in the RAM.

It is worth noting that COMSOL Multiphysics does offer hard-disk based solvers as well, but these will be slower than when data is accessed directly from the RAM. Their advantage is that they allow you to solve larger problems.

Of course, this data has to be operated on by the processors, so it turns out that the bottleneck in the solution on a desktop computer is actually the bandwidth of the memory bus — much more so than the processor speed, or even the number of processor cores.

What About Clusters?

Cluster computers are really nothing more than ordinary computers, or nodes, connected with an additional communication layer.

Let’s assume that we are working with a cluster where each node is equivalent in performance to that of our single computer. Data passes between the nodes via the interconnect hardware. The interconnect speed is dependent upon not just the type of hardware, but also the physical configuration. In sum, it is usually slower than the memory bus speed on any individual node. This introduces an additional consideration.

A simple model of a cluster with four compute nodes.

We have already seen that the partially parallelizable case is the most important to understand, so we’ll focus on that.

On a cluster, we would say that we are solving this problem in a distributed parallel sense. In the context of our game of putting together a puzzle, we can think of this as grouping several of our friends in different rooms of our house, and giving them each a stack of pieces. You would now additionally need to send pieces and information back and forth between different rooms.

COMSOL Multiphysics adjusts the solution algorithm to maximize the amount of work done locally and minimize the amount of data that is passed back and forth. These distributed parallel solvers, which are available when you use the Floating Network License, adjust the solution algorithm to efficiently split the problem up onto the different nodes of the cluster. Again, we can see that there is a limit. If there are too many nodes involved, we will just be communicating data back and forth all the time. So, for each particular problem, there is some number of nodes beyond which solution speed will not improve.

Now, if you have a completely parallel problem, such as a parametric sweep, where each step in the sweep can be solved entirely within the RAM available on one node, then a cluster is an almost perfect way to speed up your modeling. You could use up to as many nodes as there are parameter values that you want to sweep over.

Summary of Parallel Computing and COMSOL Multiphysics®

You should now have an understanding of the different types of problems that COMSOL Multiphysics solves, in terms of parallelization and how these relate to performance.

When working on a single computer, the performance bottleneck is the bus speed rather than the clock speed and number of processors. For desktop machines, we also publish some more specific hardware purchasing guidelines in our Knowledge Base. For cluster computers, performance can be much more variable, depending on problem size, cluster architecture, and the type of problem being solved. If you want more technical details about clusters, please see this series of blogs on hybrid parallel computing.

Amazon Web Services, the “Powered by Amazon Web Services” logo, and Amazon Elastic Compute Cloud are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.

JENGA® is a registered trademark owned by Pokonobe Associates.

Modeling Thermal Management: Test Set-Up

Our test set-up consists of a copper coil wound around a laminated iron core with some plastic and aluminum parts for stability. A conventional computer fan is placed one meter away from it. The occurring electromagnetic losses have to be calculated as well as the turbulent non-isothermal fluid flow around the device. The iron core has an air gap, which is intentionally included in order to analyze the influence it has on currents inside the coil and aluminum parts.

The inductor device.

Schematic of the test set-up.

First Things First

Engineers — especially those working within project deadlines — are always looking for the right balance between computational (and modeling) efforts and accuracy. Therefore, it is a good idea to start with thinking of a suitable simplification, since the aspect ratio of the model geometry is quite challenging.

The distance between the fan and device is roughly one meter, while the interior gaps between the copper winding are about 0.1 millimeters, resulting in an aspect ratio of 10,000. In order to keep the processing time as low as possible, we choose a submodeling approach. A first model with a simplified transformer geometry is used to calculate the large-scale flow field around the device. Due to symmetry, only half of the geometry is modeled. The results of this model are exported and used as an inlet condition for the following step.

Streamline plot of the velocity field. This field was used as an inlet boundary condition in the detailed model (at the position of the slice plot).

Detailed Geometry

The geometry of the detailed electrical device is built in SolidWorks® software and imported into COMSOL Multiphysics® via the CAD Import Module. Only a small part is used for the non-isothermal flow calculation in the detailed submodel (about 400 mm by 900 mm). The electromagnetic part needs to be solved for an even smaller domain (200 mm by 200 mm).

Modeling the Laminated Iron Core

The iron core is laminated in order to reduce eddy currents. We’ll use the same approach as described by TU Dresden & ABB. The material is homogenized and defined with an orthotropic electrical conductivity. This allows us to keep a single domain and a coarser mesh rather than resolving the lamination geometrically with all small plates.

Electromagnetic Losses

Due to the alternating current at 500 Hz, inductive effects in the coil (skin and proximity effect) have to be resolved. Additionally, eddy currents in the aluminum plates and iron core will heat up the device.

Surface plot of the eddy currents in the aluminum plates. The air gap in the iron core is highlighted in red. Most currents are induced close to this gap.

Slice plot of the current density inside the copper coil. The air gap within the iron core is shown in red.

Due to hysteresis, there are also some magnetization losses. These are quite small in comparison to the eddy current losses and are not explicitly solved for. The table below shows the magnetization losses as functions of the magnetic flux density Q_mag = f(B). We could use an interpolation function instead of solving hysteresis time-dependently.

Velocity Field and Temperature Distribution

The device reaches a maximum temperature of 125°C on the backside of the coil.

Streamline plot of the velocity field and surface plot of the temperature distribution.

Alternative view of the velocity field.

Best Choice for Multiphysics High Performance Computing

Today, our task was to find the best solution for computing thermal designs of transformers. In the case of BLOCK Transformatoren, they decided that COMSOL Multiphysics was the most suitable for their application after comparing the handling and results of several simulation tools.

In the end, this model involved simultaneous solving for a maximum of 8 million degrees of freedom (DOFs), using a robust combination of direct and iterative solvers. Memory (RAM) usage peaked at 89 GB of memory.

In order to be able to solve highly complex models, they chose the Ready-to-Go+ (RTG+) package with a benchmarked cluster for optimal performance. With everything being set for advanced simulations at BLOCK, we can expect their products to be pushed even closer to the limit in the future.

SolidWorks is a registered trademark of Dassault Systèmes SolidWorks Corp.

About Beowulf Clusters

In 1994, a group of researchers at NASA built a small cluster consisting of normal workstations. They called this cluster, or parallel workstation, Beowulf. Since then, the term Beowulf cluster has been used to describe clusters that are built up from commodity hardware (for example, normal workstations), using open source software. The definition is quite loose regarding the computer hardware and network interconnects. The most important point is that the workstations are no longer used as workstations, but are used as nodes in a High Performance Computing (HPC) cluster, instead.

Beowulf clusters can be used to compute all kinds of problems, but as we have mentioned earlier in the Hybrid Modeling series, in order to take advantage of the added work power in the cluster, the problem has to be parallelizable. Therefore, Beowulf clusters have been used to compute particle simulations, genetics problems, and — probably most interesting for us COMSOL Multiphysics users — parametric sweeps and large matrix multiplications.

But, why would we want to build a cluster using non-HPC hardware? One reason might be “because we already have the hardware”. For example, after an office-wide workstation or laptop upgrade has taken place, we might not know what to do with the old, outdated computers, but we still don’t want to throw them away. An alternative could be to use the concentrated computational power of idle workstations after office hours or on the weekend.

What Do We Need to Set Up the Cluster?

First of all, we need the hardware that we are going to use. For this blog post, we used our old faithful laptops as nodes, but we could just as well have used workstations or old servers. Either way, when setting up a Beowulf cluster, we should try to choose the nodes in such a way that they have similar hardware. Our laptops are no longer “performance monsters”; they are equipped with an Intel® T2400 @1.83GHz processor and 2 GB of RAM each. They are also all supplied with Ethernet network cards, so we used these to connect them together. To do this, we also need a switch. In our case, we used an old HP® 1800 switch, but, even here, we could use normal commodity hardware (such as a home office five-port switch), depending on how many nodes we are going to use.

Our Beowulf cluster, built from six old laptops and an old switch.

Since a Beowulf cluster (according to the above definition) needs an open source operating system, we installed a Linux® distribution on the laptops. Although there are specially designed operating systems for Beowulf cluster computing, it is possible to use a standard server operating system (e.g. Debian®).

When the set-up of the hardware, network, operating system, and a shared file system is done, the only step left is to install the software — COMSOL Multiphysics®. No further installation of a Message Passing Interface (MPI) or scheduler is necessary, since the COMSOL software contains all it needs in order to compute on a cluster.

Setting Up the Beowulf Cluster and Installing COMSOL Multiphysics

For our set-up, we chose Debian® Stable 6, which is one of the distributions supported by COMSOL Multiphysics at the time of writing this blog post. Next, we set up the systems. In our scenario, we tried to keep the installation as slim as possible by only installing the basic system with an additional SSH Server to get access to the cluster over the network. A desktop environment was not needed in our case; it would have reduced the performance of our Beowulf system.

After a successful installation of the operating system, we needed to set up the network and, of course, the shared file system for the compute nodes. For the shared file system, we installed the NFS server on the first node, which operates as the head node. Then, we exported the locations for the shared file system from there.

Here is one example of a set-up:

/srv/data/comsolapp     For the COMSOL application
/srv/data/comsoljobs    For the COMSOL cluster jobs storage for the users

On the compute nodes, we mounted these shares automatically.

Since we have no desktop environment installed on our systems, we need to use the automated installer (see page 77 of the COMSOL Multiphysics Installation Guide). For our purposes, we used the “setupconfig.ini” file from the installation media and edited it for our needs.

The most important step here is to set the “showgui” option to “0” instead of “1”. Another important aspect is the destination path. Here, we chose the network share because it is much easier to maintain and upgrade to new versions of COMSOL Multiphysics.

To start the installation, just add the parameter “-s /path/to/the/setupconfig.ini”, for example:

cd /media/cdrom/
./setup –s /path/to/the/setupconfig.ini

Now, the text-based installer starts and the output is sent to the terminal.

To let COMSOL Multiphysics know what compute nodes can be used, we need to write a simple “mpd.hosts” file containing the list of hostnames:

mpd.hosts
cn01
cn02
...
cn06

Finally, we start the COMSOL Server on the first node, with six nodes:

//comsol server -f mpd.hosts -nn 6 -multi on

Now you can start COMSOL Multiphysics on your desktop and connect to the server.

The Results Are in: Old Hardware, Increased Productivity

To test our “brand new” cluster, we chose a modified version of the Tuning Fork model, available in the Model Gallery. For our test run, we decided to increase the number of parameters computed in the parametric sweep to 48. We then computed the model using the COMSOL Multiphysics batch command, letting it use one to six laptops. You can see the measured total simulations per day in the graph below.

The productivity increase (Jobs/Day), taking into account the total time from opening the file to saving the result for the different number of laptops used.

As we can see, if we use six laptops, we reach almost 140 jobs per day in comparison to just under 40 jobs per day when using just one laptop. All in all, this gives us a speedup of almost 3.5. That’s impressive considering that we are using old laptops.

We have to note, though, that the measured time is not the solution time, but the total time for running the simulations. This includes opening, computing, and saving the model. Opening and saving is serial in nature, and due to Amdahl’s law (mentioned in our earlier blog post about batch sweeps), this means that we do not see the true speedup of the solver. If we were to connect to our Beowulf cluster with the COMSOL Client/Server functionality and then compare the computation times, we would obtain an even larger productivity increase compared to the numbers above.

As a conclusion, this means we can, indeed, use old hardware together with COMSOL Multiphysics to increase our productivity and speed up computations (especially parametric ones).

Debian is a registered trademark of Software in the Public Interest, Inc. in the United States.
HP is a registered trademark of Hewlett-Packard Development Company, L.P.
Intel is a trademark of Intel Corporation in the U.S. and/or other countries.
Linux is a registered trademark of Linus Torvalds.

Intro to the COMSOL API

The COMSOL API is an interface to all algorithms and data structures that define a COMSOL model. When you set up a model with the COMSOL Desktop®, you interact with the COMSOL API behind the scenes. LiveLink™ for MATLAB®, the topic of a recent blog post, also operates using the COMSOL API but in an interactive fashion rather than compiled. Today, we focus on the COMSOL API for use with Java®.

COMSOL Desktop as Code Generator

You don’t have to be an expert Java® programmer to get started with the COMSOL API. You can dive right in from the tool you are already working with — the COMSOL Desktop. Each action you perform in this graphical modeling environment gets recorded in the model history. You can export this history as Java code by saving your model and choosing “Model File for Java®” as the file type. This method is useful to create building blocks for your program.

Hello World!

Let’s start with a simple example to get familiar with the process: the COMSOL API version of a “Hello, World!” program.

In the COMSOL Desktop, create a model that only contains a 3D geometry. Add a block to the geometry with dimensions of 0.1m x 0.2m x 0.5m and save a “Model File for Java®” with the name “HelloWorld.java”.

If you open the output in a text editor, it will look like this:

import com.comsol.model.*; 
import com.comsol.model.util.*;

public class HelloWorld {

   public static void main(String[] args) {
      run(); 
   }
   public static Model run() {
      Model model = ModelUtil.create("Model"); 
      model.modelNode().create("comp1"); 
      model.geom().create("geom1", 3); 
      model.geom("geom1").feature().create("blk1", "Block"); 
      model.geom("geom1").feature("blk1").set("size", new String[]{"0.1", "0.2", "0.5"});
      model.geom("geom1").run("fin");
      return model; 
   } 
}

The first two lines are import statements that point to the COMSOL API. This is followed by the definition of the class HelloWorld. The class name is the same as the file name, as is the rule in Java programming.

The class contains a main() method that in turn calls a static run() method to build and return a Model object. For small programming projects, you can directly modify this method. In other words, you don’t necessarily have to use the more advanced object-oriented features of Java.

The Compact History Function

The COMSOL Desktop offers another feature that is useful when generating code: the “Compact History” function in the File menu. When you set up a model, you usually add some features that you later remove or move around. All of these modifications get recorded in the model history, which then holds many unnecessary steps.

The “Compact History” function cleans up the history, removes duplicate and deleted entries, and reorders everything according to the order in the Model Builder. If you use the function before exporting, you will get clean code.

Why not automatically clean up the history before saving a Java® file then? Well, it is sometimes useful not to compact the history.

Suppose you are developing code that uses the COMSOL API and it turns out you need some parts that you can quickly set up with the COMSOL Desktop. You start modifying the model you have been working with and save it as a Java file. Fortunately, you have not compacted the model history before saving and can therefore easily find all the changes right at the end of the exported code. That is much easier than trying to spot changes otherwise scattered throughout the entire model code.

Compiling and Running COMSOL API Code

Java® is a compiled language and before you can do useful things with your code, you have to compile it in a class file. For this you need a Java compiler, such as the free Java® Development Kit (JDK).

Once you have installed a JDK, you can use the

comsolcompile ( comsol compile on Linux® or Mac®)

command, which is part of a COMSOL software installation, to compile your code. The command automatically sets up the path to the COMSOL API for the Java® compiler.

To compile the example above, use the command

comsolcompile -jdkroot PATH_TO_JDK HelloWorld.java

Here, PATH_TO_JDK is the directory where you have installed the JDK. Note that the COMSOL API adheres to Java® 1.5 and the above method works with JDK 1.5 or 1.6.

You can also use an Integrated Development Environment (IDE), such as Eclipse™. Set up your projects with Java 1.5 compatibility and add all JAR files in the “plugins” directory of the COMSOL Multiphysics® installation to your build path.

Once you have compiled your code into a class file, you can open it in the COMSOL Desktop via the File > Open menu. If you do that with the example above, you will see your “Hello World” model with one block in a 3D geometry. The same result can be achieved with a regular COMSOL mph model file. Next, we want to change that and do something beyond the capabilities of a regular model file.

But, before we go ahead and modify the example, let’s examine the structure and meaning of the code in the run() method above.

The COMSOL API in a Nutshell

The “Hello World” example can already teach us some of the most important aspects of working with the COMSOL API. Let’s go through the run() method to understand what’s happening.

The first line,

Model model = ModelUtil.create("Model");

creates a new model with ModelUtil.create(), — a static method that takes a name (the String "Model") as an argument. ModelUtil is the COMSOL API’s little helper, a collection of utility methods. You can use this to load models or create new ones from scratch, for example.

ModelUtil.create()

returns a Model object. This object holds all the settings of a COMSOL model, i.e. it contains the entire model tree that you usually see in the Model Builder of the COMSOL Desktop.

The line

model.modelNode().create("comp1");

creates a new component node in the model tree. A model component has an associated geometry that is added with the line

model.geom().create;("geom1", 3);

The second argument (the numeral 3) makes the component geometry 3D.

Note that the first argument for both create() methods is a String, the so-called tag. Tags are used everywhere in the model object to uniquely identify the features that are contained in the model. This is necessary because a model can contain many features of the same type. You can, for instance, have more than one component in your model, each with its own geometry. The component geometry might contain many primitive shapes of the same type. The physics settings might then use many boundary conditions of the same type, and so on. Thus, giving each item a unique tag is a way of housekeeping.

You can make the tags of any COMSOL Multiphysics model visible in the COMSOL Desktop by enabling the “Show Name and Tag” or “Show Type and Tag” setting in the “Model Builder Node Label” menu of the Home tab.

Tags are displayed in the Model Builder of the COMSOL Desktop when the “Show Name and Tag” or “Show Type and Tag” option is selected from the Model Builder Node Label settings.

The next line of code,

model.geom("geom1").feature().create("blk1", "Block");;

creates a block in the first geometry "geom1".

You can recognize the hierarchy of the model tree in this line. The first part, model.geom("geom1") associates the command with geometry "geom1" and the second part, feature().create("blk1", "Block") adds a new feature to it. The feature is a block identified with the tag "blk1". Thinking in terms of the COMSOL Desktop, you can picture the first part as a right-click on "geom1", and the second part as choosing Block from the geometry menu that pops up.

After the block is created, its properties are modified. This happens in the line

model.geom("geom1").feature("blk1").set("size", new String[]{"0.1", "0.2", "0.5"});

Again, the first part identifies the first block "blk1" of "geom1", and the second part modifies the size property with a set() method.

The first argument of the latter specifies the property that you want to change, “size” in this case. The second argument assigns the new values — in this particular case that would be the width, height, and length properties of the block.

Note, that although these properties are set to real numbers, the argument is passed as an array of strings. Why is that? Remember, that in the COMSOL software, you can type in a mathematical expression instead of specific numbers anywhere you want. This is also true for the COMSOL API, and for that reason such properties are passed as strings.

Finally, before the new model is returned, the line

model.geom("geom1").run("fin");

runs the finalization of the geometry, just like when you press the “Build All” button in the COMSOL Desktop.

This is the COMSOL API in a nutshell and all you need to know to get started. There are, of course, many more details, but you can always pick them up as you go along, by using the COMSOL Desktop as a code reference together with the documentation of the COMSOL API for use with Java®.

Example: Building a Spiral Inductor geometry

To illustrate an application of the COMSOL API, we can look at the model of a spiral inductor from our Model Gallery.

The aim of the model is to compute the self inductance for the given coil design. If you look at the geometry closely, you can see that it is built up from an arrangement of blocks. In the model, the dimensions of the spiral are fixed to particular values. When designing such a device, it is useful to look at different configurations, like inductors with different cross sections and number of windings, for instance. To this end, you can set up the geometry parametrically in the COMSOL Desktop.

Parameterizing the cross section is easy to do there. The number of windings, on the other hand, cannot be parameterized in the COMSOL Desktop, because the length of a piece of wire changes every third turn. As you saw above, however, you can use the COMSOL API to specify all properties of a block in your Java® program. You can use that to build a spiral conductor geometry automatically.

From the “Hello World” example above, you already know how to build one block. To build a spiral conductor, you create many blocks with different size properties and orientations and arrange them in the desired configuration. In the code, you need a couple of variables to keep track of:

You then generate the blocks that make up the spiral in a loop.

As you now know, you need a unique tag to identify each block. You could create that yourself, by concatenating a block counter to a base string, such as "blk". The better alternative, however, is to use the uniquetag() method provided by the COMSOL API. This method does the same thing, but keeps track of counters internally and makes sure that a tag is not used twice.

In the code snippet below, the tag for each block is generated with

model.geom("geom1").feature().uniquetag("blk")

With the new unique tag, you now create a block and set its properties. Aside from the "size", you also have to change the "pos" and "rot" properties for the position and orientation respectively. After that, you update the variables for the next iteration.

A spiral inductor geometry is generated automatically with the COMSOL API for use with Java®.

What’s Next?

You can use the COMSOL API for many more things than creating geometries, of course. In fact, you can automate any modeling task you normally perform with the COMSOL Desktop. In the example of the spiral inductor, you could, for instance, continue to compute the new result of the updated geometry. You could also run a parametric sweep in your code to compute a whole range of spiral conductors automatically. Then you could proceed to process the results of the computation by creating plots that are exported to image files, extracting the inductance, and exporting all values to a file.

We have merely scratched the surface of the capabilities of the COMSOL API. Beyond programming the tasks you can do manually in the COMSOL Desktop, the COMSOL API gives you access to and control over data structures like the finite element mesh, finite element matrices, and the solution data sets.

Besides writing class files that can be opened by the COMSOL Desktop, you can write programs that connect to a COMSOL Server process or even standalone programs that integrate COMSOL technology. Therefore, if you routinely deal with a particular simulation task, the COMSOL API is a powerful and flexible tool for you to automate that task.

Eclipse is a trademarks of Eclipse Foundation, Inc. Linux is a registered trademark of Linus Torvalds. Mac is a trademark of Apple Inc., registered in the U.S. and other countries. MATLAB is a registered trademark of The MathWorks, Inc. Oracle and Java are registered trademarks of Oracle and/or its affiliates.

Amdahl’s and Gustafson-Barsis’ laws

We have mentioned before how speedup through the addition of compute units is dependent on the algorithm (in this post we will use the term processes, but added compute units can also be threads). A strictly serial algorithm, like computing the elements of the Fibonacci series, does not benefit at all from an added process, while a parallel algorithm, like vector addition, can make use of as many processors as we have elements in the vector. Most algorithms in the real world are somewhere in between these two.

To analyze the possible maximum speedup of an algorithm, we will assume that it consists of a fraction of perfectly parallelizable code and a fraction of strictly serial code. Let us call the fraction of parallelized code , where is a number between (and including) 0 and 1. This automatically means that our algorithm has a fraction of serial code that is equal to .

Considering the computation time, , for active processes, and starting with the case , we can use the representation . When running processes, the serial fraction of the code is not affected, but the perfectly parallelized code will be computed times faster. Therefore, the computation time for processes is , and the speedup is .

Amdahl’s Law

This expression is at the heart of Amdahl’s law. Plotting for different values of and , we now see something interesting in the graph below.

The speedup for increasing the number of processes for different fractions of parallelizable code.

For 100% parallelized code, the sky is the limit. Yet, we find that for , the asymptotic limit or the theoretical maximal speedup is .

For a 95% parallelized code, we find that — a maximum speedup of twenty times, even if we have an infinite number of processes. Furthermore, we have , , and . The theoretical maximum speedup decreases quickly when decreasing the fraction of parallelized code.

But don’t give up and go home just yet!

Gustafson-Barsis’ Law

There is one thing that Amdahl’s law does not consider, and that is the fact that when we buy a faster and larger computer to be able to run more processes, we usually don’t want to compute our small models from yesterday faster. Instead, we want to compute new, larger (and cooler) models. That’s what the Gustafson-Barsis’ law is all about. This is based on the assumption that the size of the problem we want to compute increases linearly with the number of available processes.

Amdahl’s law assumes that the size of the problem is fixed. When adding new processors they are working on parts of the problem that was originally handled by a lesser number of processes. By adding more and more processes, you are not utilizing the full ability of the added processes as eventually the size of what they are able to work on reaches a lower limit. Yet, by assuming that the size of the problem increases with the number of added processes, then you are utilizing all the processes to an assumed level, and the speedup of the performed computations remains unbounded.

The equation describing this phenomenon is , which gives us a far more optimistic result for what is called scaled speedup (which is like productivity), as shown in the graph below:

When taking into account that the size of the job normally increases with the number of available processes, our predictions are more optimistic.

The Cost of Communication

Gustafson-Barsis’ law implies that we are only restricted in the size of the problem we can compute by the resources we have for adding processes. Yet, there are other factors that affect speedup. Something we’ve tried to stress so far in this blog series is that communication is expensive. But we haven’t talked about how expensive it is yet, so let’s look at some examples.

Let’s consider an overhead that is dominated by the communication and synchronization required in parallel processing, and model this as time added to the computation time. This means that the amount of communication increases when we increase the number of processes, and that this increase will be modeled by a function , where is a constant and is some function. Hence, we can compute the speedup through: .

The graph below shows the case where the fraction of parallelized code is 95%, and where we can see the speedup for an increasing number of processes, for different functions of , assuming = 0.005 (this constant would vary between different problems and platforms). In the case of no overhead, the result is as predicted by Amdahl’s law, but when we start adding overhead, we see that something is happening.

For a linearly increasing overhead, we find that the speedup doesn’t reach a value greater than five before the communication starts to counteract the increased computation power added by more processes. For a quadratic function, , the result is even worse and, as you might recall from our earlier blog post on distributed memory computing, the increase of communication is quadratic in the case of all-to-all communication.

Speedup with added overhead. The constant, c, is chosen to be 0.005.

Due to this phenomenon, we cannot expect to have a speedup on a cluster for, say, a small time-dependent problem when adding more and more processes. The amount of communication would increase faster than any gain from added processes. Yet, in this case, we have only considered a fixed problem size and the “slowdown” effect introduced through communication would be less relevant as we increase the size of our problem.

Batch Sweeps in COMSOL Multiphysics

Let us now leave the theoretical aspect and learn how to make use of the batch sweep feature in COMSOL Multiphysics. As our example model, we will use the electrodeless lamp, which is available in the Model Gallery. This model is small, at around 80,000 degrees of freedom, but needs about 130 time steps in its solution. To make this transient model parametric as well, we will compute the model for several values of the lamp power, namely 50 W, 60 W, 70 W, and 80 W.

On my workstation, a Fujitsu® CELSIUS® equipped with an Intel® Xeon® E5-2643 quad core processor and 16 GB of RAM, the following compute times are received:

The speedup here is far from perfect — just about 1.7 for three cores and has even decreased for four cores. This is due to fact that it is a small model with a low number of degrees of freedom per thread within each time step.

We will now use the batch sweep functionality to parallelize this problem in another way: we will switch from data parallelism to task parallelism. We will create a batch job for each parameter value and see what this does to our computation times. To do this, we first activate the “Advanced Study Options”, then we right-click on “Study 1” and choose “Batch Sweep”, as illustrated in the animation below:

How to activate batch sweep in a model, include the parameter values, and specify the number of simultaneous jobs.

The graph below indicates the productivity or “speedup” we can get by controlling the parallelization. When running one batch job using four cores, we get the result from above: 72 minutes. When changing the configuration to two batch jobs simultaneously, each using two cores, we can compute all the parameters in 48 minutes. Finally, when computing four batch jobs at the same time, each using one processor, the total computation time is 34 minutes. This gives speedups of 2.5 and 3.5 times, respectively — a lot better compared to parallelizing through using pure shared memory alone.

Simulations per day for the electrodeless lamp model. “4×1″ means four batch jobs run simultaneously, using one core each.

Concluding the Hybrid Modeling Series

Throughout this blog series, we have learned about shared, distributed, and hybrid memory computing and what their weaknesses and strengths are, as well as the large potential of parallel computing. We have also learned that there is no such thing as a free lunch when it comes to computing; we cannot just add processes and hope for perfect speedup for all types of problems.

Instead, we need to choose the best way to parallelize a problem to get the most performance gain out of our hardware, much like we have to choose the correct solver to get the best solution time when solving a numerical problem.

Selecting the right parallel configuration is not always easy, and it can be hard to know beforehand how you should “hybridize” your parallel computations. But as in many other cases, experience comes from playing around and testing, and with COMSOL Multiphysics, you have the possibility to do that. Try it yourself with different configurations and different models, and you will soon know how to set the software up in order to get the best performance out of your hardware.

Fujitsu is a registered trademark of Fujitsu Limited in the United States and other countries. CELSIUS is a registered trademark of Fujitsu Technology Solutions in the United States and other countries. Intel and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.

Introducing Hybrid Computing

In recent years, cluster systems have become more and more powerful by incorporating the latest multicore technologies. Parallelism now spreads out across several levels. Huge systems have to deal with parallelism across nodes, sockets, cores, and even vector units (where operations are performed on short vectors or 1D arrays of data, not on scalar values or single data items).

In addition, the memory systems are organized into several levels as well. And as these hierarchies get deeper and deeper and more complex, the programming and execution models need to reflect these nested configurations. It turns out that it is not sufficient to deal with just a single programming and execution model. Therefore, computing is becoming increasingly hybrid.

Core’s Law and Clusters

Multiple computing cores are ubiquitous and everyone needs to deal with parallelism. As clock frequencies have stalled at a high level of around 2-5 GHz, the ever increasing hunger for computing power can only be satisfied by adding more and more cores. The well-known Moore’s law has turned into a corollary Core’s law — stating that the core count per die area will keep on increasing exponentially.

A direct consequence of this development is that the resources per core (e.g. the cache memory and number of memory channels) will become smaller due to logistics. The latest multicore incarnations, of the classical CPU-type, have up to sixteen cores but only up to four memory channels.

Typically, several multicore CPUs are providing highly capable shared memory nodes that bring impressive computing power in terms of GigaFLOP/s (i.e. billion floating point operations per second). Several of these shared memory nodes are then integrated into clusters via high-speed interconnects — providing nearly unlimited resources in terms of (Giga/Tera/Peta)FLOP/s and memory capacities. The only limits are your IT department’s budget and floor space.

These days, a cluster system needs to have more than 100,000 cores in order to receive a high ranking in the recent TOP500 list.

Reaching the Limits

A cluster represents a distributed memory system where messages are sent between the nodes by means of message passing. Here, the Message Passing Interface (MPI) with several open source and commercial implementations is the defacto standard. Typically, inside the node, OpenMP is used for shared memory programming.

Yet, numerical experiments easily unveil the limitations of multicore platforms: it is getting more and more difficult to feed the beasts. Put plainly, it is hard to get as much data to the cores quickly enough to keep them busy and crunching numbers. Basically, you could say that FLOP/s are available for free but you must keep an eye on the computational intensity, i.e. the number of FLOP per data element — a characteristic of the algorithm. The computational intensity is providing an upper bound for the achievable FLOP/s-rate.

If the computational intensity is increasing linearly in problem size, the bandwidth is not limiting performance. But the typical operation for finite element numerical simulations is of sparse matrix-vector type, which is bandwidth-bound, and the bandwidth is typically proportional to the number of memory channels in a multicore system. Once the available memory bandwidth is saturated, turning on more and more cores has no added value. This is the reason why speedups on multicore CPUs and shared memory platforms often saturate as the memory traffic is already at a maximum level, even though not all cores are being used for computations. Cluster systems, on the other hand, have the additional benefit of increased accumulated bandwidth and therefore better performance. (For further details about bandwidth limitations exemplified by the STREAM benchmark go here.)

On the hardware side, attempts have been made to mitigate the bandwidth limitations by introducing a hierarchy of caches. These can range from small level-1 caches, restricted to a single core with only a few hundred KB of memory size, to up to big level-3 caches, shared between several cores with up to a few dozen MB of memory size. The aim of the caches is to keep the data as close to the cores as possible, so that data that is to be reused does not need to be transferred from the main memory over and over again. This removes some of the pressure from the memory channels.

Even a single multicore processor itself can bring about a nested memory hierarchy. Yet, packaging several multicore processors into multiple sockets builds a shared memory node with non-uniform memory access (NUMA). In other words, some parts of the application data is stored in memory local to a core and some of the data is stored in remote memory places. Therefore, some parts of the data can be accessed very fast while other accesses have longer latencies. This means that the correct placement of data and a corresponding distribution of compute tasks are critical for performance.

Being Aware of Hierarchies on Performance

We have learned that shared memory systems build up a hierarchical system of cores and memories and that the programming model, algorithms, and implementations need to be fully aware of these hierarchies. As the computing resources of a shared memory node are limited, additional power can be added by connecting several shared memory nodes by a fast interconnection network in distributed computing.

To bring back our previous analogy from the shared memory and distributed memory computing blog posts, we are now using a variable number of conference locations to represent the cluster, where each location provides a conference room with a big table to represent a shared memory node.

If the overall work to be done increases more and more, the conference manager can call other locations of the company for help. Suppose the conference rooms are located in Boston, San Francisco, London, Paris, and Munich, for instance. These remote locations represent the distributed memory processes (the MPI processes). The manager can now include new locations on demand, such as adding Stockholm — or in terms of hybrid computing, she can set up additional processes (i.e. conference tables) per shared memory node (i.e. per conference room location).

Each conference room location (process) has a phone on the table in the meeting room, which employees can use to call any other location (another process) and ask for data or information (message passing). The local staff (a limited resource) is sitting around each conference table in a particular location. Every employee at the conference table represents a thread that helps solving the tasks at the conference room table.

On the table, local data is available in a report (level-1 caches), in folders (level-2 caches), in folders located inside cabinets (level-3 caches), in the library on the same floor (main memory), or are filed in the archive in the basement (hard-disk). Several assistants (memory channels) are running around in the building in order to fetch new folders with requested information from the library or the archive. The number of assistants is limited and they can only carry a limited number of folders at the same time (bandwidth).

Having more people at the table does not add any value if there are no more assistants available who can bring them enough data to work with. It is clear that the conference manager needs to make sure that the data necessary for the work is available on the table and that all the employees in the room can contribute effectively to reach a solution to a given problem. She should also make sure that the number of calls to other conference locations via the phone on the table is kept at a minimum. In terms of numerics, the implementations should be hierarchy-aware, data should be kept local, and the amount of communication should be kept at a minimum.

The phone calls between conference locations represent MPI calls between processes. On the meeting room table, a shared memory mechanism should be employed. In total, a perfect interplay of distributed memory (MPI) and shared memory (OpenMP) is required.

Example of a Hybrid Cluster Configuration

Let’s take a closer look at some possible cluster and core configurations. In our test benchmark model below, we investigated a small cluster that is made up of three shared memory nodes. Every node has two sockets with a quad-core processor in each socket. The total core count is 24. Each processor has a local memory bank, also illustrating the NUMA configuration of main memory.

Now, we test the cases where three, six, twelve, or twenty-four MPI processes are configured on this cluster. With three MPI processes, we have one MPI process per node and eight threads per MPI process that can communicate via shared memory/OpenMP inside the node and across the two sockets of the node. With six MPI processes we would have one MPI process per socket, i.e. one per processor. Each MPI process then needs four threads. The third possibility of twelve MPI processes is to set up two MPI processes per processor with two threads each. Finally, we can test one MPI process per core totaling to twenty-four MPI processes on the system. This is the non-hybrid case, where no shared memory parallelism is needed, and all communication occurs through distributed memory.

Which configuration do you think is the best one?

Different MPI configurations on a cluster with three shared memory nodes consisting of two sockets, each with a quad-core processor and local memory banks.

Why Hybrid?

Why not use a single programming and execution model and ignore the hierarchical core and memory configuration? First of all, it’s because shared memory (OpenMP) mechanisms cannot be used globally on standard type systems with standard installations (no world-spanning table is available to share the data).

So why not use message-passing globally across all cores — with 24 MPI processes as in the example above? Of course it would be possible; you can use message-passing even between cores on the same shared memory node. But it would mean that every employee, in our analogy above, would have his or her own phone and would have to make calls to all other employees worldwide. There would be problems of engaged signals or people put on hold and, as a result, not working.

In fact, the real scenario is more complex because modern MPI implementations are well aware of hierarchies and are trimmed to use shared memory mechanisms for local communication. One of the downsides of MPI is that memory resources are wasted quadratically in the number of participating processes. The reason for this is that internal buffers are set up where the data is stored (and possibly duplicated) before the actual messages can be sent between the processes. On 10⁶ cores, this would require 10¹² memory buffers for a single global communication call (if the MPI implementation is not hierarchy-aware). In hybrid computing, the number of MPI processes is typically lower than the core count — saving resources in terms of memory and data transfers.

Another big advantage of using the hybrid model is that many mechanisms (like data placement, thread pinning, or load balancing, to name a few) need some dedicated actions of the programmer. The hybrid model provides a much more versatile tool to express these details. It shares the advantages of a global and discrete view of the memory. It is a natural choice to consider the hybrid OpenMP + MPI model since it fits with the hybrid structure of memory and core configurations, as illustrated in the figures.

Most importantly, the hybrid model is flexible and adaptable and helps reduce overheads and demands for resources. In terms of finite element modeling, hierarchies in the data and tasks can often be derived from the physical model, its geometry, the algorithms, and the solvers used. These hierarchies can then be translated into shared and distributed memory mechanisms.

Of course, the hybrid model also combines the pitfalls of both shared memory and distributed computing, and ends up being much more complex. However, the final outcome is worth the effort! COMSOL Multiphysics provides sophisticated data structures and algorithms that represent and exploit multi-level parallelism to a large extent. It supports shared memory and distributed memory at the same time and the user can tune the interaction of both by a set of parameters for best performance.

Benchmarking Your Models and Your System

After all these theoretical explanations, it’s now time to connect the concept with real models. I suppose you are quite curious to investigate the scalability, the speedups, and the productivity gains when running COMSOL Multiphysics in parallel on the compute servers and clusters in your department.

In order to obtain proper results, it is very important to keep an eye on the problem size. The size of the subproblems on a shared memory node have to be large enough for every thread to obtain a reasonable amount of work and that the ratio of computations per process to the amount of data exchanged via messages between the processes is sufficiently large. As Pär had mentioned in his earlier blog post, it is crucial to consider whether problems are parallelizable at all. For example, if the major effort when setting up your model is dedicated to computing a long time-stepping series (that may last for hours) but the problem size in every time step is not significantly large, you will not see significant benefits when ramping up additional nodes and cores.

I encourage you to try various configurations of the hybrid model, even when you only have a shared-memory machine available.

A Hybrid Scalability Study

In the test scenario presented here, we consider a structural mechanics model representing a ten-spoke wheel rim where the tire pressure and the load distribution is simulated.

Model of a wheel rim and its corresponding submodel.

Our simulation is run on the three compute nodes mentioned above, where each node has two sockets with a quad-core Intel Xeon® E5-2609 processor with 64 GB RAM per node, and where 32 GB are associated with a processor. The nodes are interconnected by (a rather slow) Gigabit ethernet connection. In total, we have 24 cores available on this particular machine.

In the graph below, we compare the number of simulations of this one model that can be run per day, according to the hybrid model configuration. We consider the case of 1, 2, 3, 6, 12, and 24 MPI processes running. This leads to 1, 2, 3, 4, 6, 8, 12, 16, and 24 active cores running, depending on the configuration. Each bar in the graph represents an (nn x np)-configuration, where nn is the number of processes, np is the number of threads per process, and nn*np is the number of active cores. The bars are grouped into blocks with the same active core count, where their configuration is listed at the top of the bars.

The graph shows that the performance increases, in general, with the number of active cores. We see slight variations for the different configurations. When reaching the full system load with 24 active cores, we find that the best configuration is to assign one MPI process per socket (i.e. six MPI processes in total). The performance and productivity gain on the three-node system with a hybrid process-thread configuration (case 6×4) is more than twice as good as the performance on a single shared memory node (case 1×8). It is also almost 30% better than the completely distributed model (case 24×1), which uses the same number of cores.

When comparing this to its neighboring fully-distributed configuration (case 12×1), we see that there is no real gain in performance despite doubling the number of cores. This is because the slow Gigabit ethernet network is already very close to its limits with 12 MPI processes. More MPI processes are therefore not beneficial. The situation is different when comparing the 12×1- and 12×2-configurations, where the number of threads per process is doubled along with the number of active cores. This basically means that, in this case, the amount of communication via ethernet is not increased.

Benchmarking a structural mechanics model of a wheel rim using different configurations in a hybrid model. The y-axis indicates performance and productivity gain through the total number of simulations of this model that can be run during a day. The bars indicate different configurations of nn x np, where nn is the number of MPI processes, and np is the number of threads per process.

Setting up Hybrid Runs in COMSOL Multiphysics

When running COMSOL Multiphysics in the parallel hybrid mode, you have various possibilities to adjust the number of processes and threads used. Some settings can be found in the Multicore and Cluster Computing section in the Preference dialogue or in the Cluster Computing subnode of the Study node in the Model Builder. You can fine-tune the settings in the Cluster Computing subnode of the Job Configurations node, where you can specify the number of (physical) nodes you want to use in your cluster computation. Using the drop-down menu in the Settings window, you have additional choices, like the number of processes per host or the node granularity, which decides whether to put one process per node, socket, or core.

The number of threads used per process is set automatically. COMSOL Multiphysics always utilizes the maximum number of available cores, i.e. the number of cores per process is set to the number of available cores on the node divided by the number of processes on the node. You can override the number of cores per process by setting the Number of processors in the Multicore section of the Preferences dialogue to the requested value.

On Linux systems, you have the command line options -nn for the number of processes, -np for overwriting the automatically determined number of threads per process, and the option -nnhost that sets the number of processes per host. A natural choice for nnhost is either 1 or the number of sockets per node.

Next Steps

• For additional options, use cases, and examples please refer to the COMSOL Multiphysics documentation.
• Our final post in this Hybrid Modeling blog series will be on the topic of batch sweeps.

Processes and Clusters

Recalling what we learned in the last blog post, we now know that shared memory computing is the utilization of threads to split up the work in a program into several smaller work units that can run in parallel within a node. These threads share access to a certain portion of memory — hence the name shared memory computing. By contrast, the parallelization in distributed memory computing is done via several processes executing multiple threads, each with a private space of memory that the other processes cannot access. All these processes, distributed across several computers, processors, and/or multiple cores, are the small parts that together build up a parallel program in the distributed memory approach.

To put it plainly, the memory is not shared anymore, it is distributed (check out the diagram in our first blog post in this series.)

To understand why distributed computing was developed in this way, we need to consider the basic concept of cluster computing. The memory and computing power of a single computer is limited by its resources. In order to get more performance and to increase the amount of available memory, scientists started connecting several computers together into what is called a computer cluster.

Splitting of the Problem

The idea of physically distributing processes across a computer cluster results in a new level of complexity when parallelizing problems. Every problem needs to be split into pieces — the data needs to be split and the corresponding tasks need to be distributed. Consider a matrix type problem, where operations are performed on a huge array. This array can be split into blocks (maybe disjointed, maybe overlapping) and each process then handles its private block. Of course, the operations and data on each block might be coupled to the operations and data on other blocks, which makes it necessary to introduce a communication mechanism between the processes.

To this end, data or information required by other processes will be gathered into chunks that are then exchanged between processes by sending messages. This approach is called message-passing, and messages can be exchanged globally (all-to-all, all-to-one, one-to-all) or point-to-point (one sending process, one receiving process). Depending on the couplings of the overall problem, a lot of communication might be necessary.

The objective is to keep data and operations as local as possible in order to keep the communication volume as low as possible.

The number of messages that needs to be sent in all-to-all can be described by a complete graph. The increase is quadratic with respect to the number of compute nodes used.

Speeding up Computations or Solving Larger Problems

The scientist who is working on a computer cluster can benefit from the additional resources in the following two ways.

First, with more memory and more computing power available, she can increase the problem size and hereby solve larger problems in the same amount of time through adding additional processes and keeping the work load per process (i.e. size of the subproblem and number of operations) at the same level. This is called weak scaling.

Alternatively, she can maintain the overall problem size and distribute smaller subproblems to a larger number of processes. Every process then needs to deal with a smaller workload and can finish its tasks much faster. In the optimal case, the reduction in computing time for a problem of fixed size distributed on P processes will be P. Instead of one simulation per time unit (one hour, one day, etc.), P simulations can be run per time unit. This approach is known as strong scaling.

In short, distributed memory computing can help you solve larger problems within the same amount of time or help you solve problems of fixed size in a shorter amount of time.

Communication Needed

Let’s now take a closer look at message-passing. How do the processes know what the other parts of the programs are doing? As we know from above, the processes explicitly have to send and receive the information and variables that they or other processors need. This, in turn, brings with it some drawbacks, especially concerning the time it takes to send the messages over the network.

As an illustration, we can recall the conference room analogy discussed in the shared memory blog post, where collaborative work occurred around a table and all the information was made freely available for everyone sitting at that table to access and work in, even in parallel. Suppose that this time, the conference room and its table have been replaced by individual offices, where the employees sit and make changes to the papers they have in front of them.

In this scenario, one employee, let’s call her Alice, makes a change to report A, and wants to alert and pass these changes to her coworker Bob. She now needs to stop what she’s doing, get out of her office, walk over to Bob’s office, deliver the new information, and then return to her desk before continuing to work. This is much more cumbersome than sliding a sheet of paper across the table in a conference room. The worst case in this scenario is that Alice will spend more time alerting her coworkers about her changes than actually making changes.

The communication step in the new version of our analogy can be a bottleneck, slowing down the overall work progress. If we were to reduce the amount of communication that needs to be done (or speed up the communication, perhaps by installing telephones in the offices, or using an even faster kind of network) we could spend less time waiting on messages being delivered and more time computing our numerical simulations. In distributed memory computing, the bottleneck is usually the technology that passes electronic data to each other, the wires between the nodes, if you like. The current industry standard for providing high throughput and low latency is Infiniband, which allows the passage of messages to occur a lot quicker than ethernet.

Why Use Distributed Memory?

Distributed memory computing has a number of advantages. One of the reasons why you would utilize distributed memory is the same as in the shared memory case. When adding more compute power, either in the form of additional cores, sockets, or nodes in a cluster, we can start more and more processes and take advantage of the added resources. We can use the gained compute power to get the results of the simulations faster.

With the distributed memory approach, we also get the advantage that with every compute node added to a cluster, we have more memory available. We are no longer limited by the amount of memory our mainboard allows us to build in and we can, in theory, compute arbitrarily large models. In most cases, scalability of distributed memory computing exceeds that of shared memory computing, i.e. the speed-up will saturate at a much larger number of processes (compared to the number of threads used).

Number of simulations per day with respect to the number of processes used for the Perforated Muffler model depicted below. A 1 GB/s ethernet was used as the communication network. The first four processes are executed on one compute node, and the ethernet network comes into use after that. The small difference in simulations per day between 4 and 5 processes shows the impact of a slow communications network, even for a parametric problem. The compute nodes used are equipped with Intel® Xeon® E5-2609 and 64 GB DDR3 @1600 MHz.

However, we do have to be aware of the limitations as well. Just as in the shared memory case, there are some problems that are well suited for computing with the distributed memory approach, while others are not. This time, we also need to look at the amount of communication that is needed to solve the problem, not only if it is easily parallelized.

Consider, for instance, a time-dependent problem where a large amount of particles are interacting in such a way that after each step, all of the particles need to have the information about every other particle. Assuming that each of the particles are computed by its own process, the amount of communications in such an example can be described by the fully connected graphs (shown above), and the number of messages per iteration grows rapidly as the number of particles and processes increase. In contrast, a parametric sweep, where the parametric values can be computed independently of each other, requires almost no communication at all, and will not suffer as much from the communication bottleneck.

The model for which the speed-up was obtained. It is a small parametric model (750,000 degrees of freedom) using the PARDISO direct solver. This model is available in the Model Gallery.

How COMSOL Takes Advantage of Distributed Memory Computing

Users that have access to a floating network license (FNL) have the possibility to use the distributed functionality of the COMSOL software on single machines with multiple cores, on a cluster, or even in the cloud. The COMSOL software’s solvers can be used in distributed mode without any further configuration. Hence, you can do larger simulations in the same amount of time or speed up your fixed-size simulation. Either way, COMSOL Multiphysics helps you to increase your productivity.

The distributed functionality is also very useful if you are computing parametric sweeps. In this case, you can automatically distribute the runs associated to different parameter values across the processes you are starting COMSOL Multiphysics with. Since the different runs in such a sweep can be computed independently of each other, this is called an “embarrassingly parallel problem”. The speed-up is, with a good interconnection network, almost the same as the number of processes.

For a detailed instruction on how to set up your compute job for distributed memory computing, we recommend the COMSOL Reference manual, where we list several examples on how to start COMSOL in distributed mode. You may also refer to the user guide of your HPC cluster for details about how to submit computing jobs.

Next up in this blog series, we will dig deeper into the concept of hybrid modeling — check back soon!