When converting a UDF to run in parallel, some parts of the function may need to be done by the host and some by the compute nodes. This distinction is made when the UDF is compiled. By using Ansys Fluent-provided compiler directives, you can specify portions of your function to be assigned to the host or to the compute nodes. The UDF that you write will be written as a single file for the host and node versions, but different parts of the function will be compiled to generate different versions of the dynamically linked shared object file libudf.so (libudf.dll on Windows). Print tasks, for example, may be assigned exclusively to the host, while a task such as computing the total volume of a complete mesh will be assigned to the compute nodes. Since most operations are executed by either the host or compute nodes, negated forms of compiler directives are more commonly used.

Note that the primary purpose of the host is to interpret commands from Cortex and to pass those commands (and data) to compute node-0 for distribution. Since the host does not contain mesh data, you will need to be careful not to include the host in any calculations that could, for example result in a division by zero. In this case, you will need to direct the compiler to ignore the host when it is performing mesh-related calculations, by wrapping those operations around the #if !RP_HOST directive. For example, suppose that your UDF will compute the total area of a face thread, and then use that total area to compute a flux. If you do not exclude the host from these operations, the total area on the host will be zero and a floating point exception will occur when the function attempts to divide by zero to obtain the flux.

Example

#if !RP_HOST avg_pres = total_pres_a / total_area; /* if you do not exclude the host
    this operation will result in a division by zero and error!   
   Remember that host has no data so its total will be zero.*/
 #endif 

You will need to use the #if !RP_NODE directive when you want to exclude compute nodes from operations for which they do not have data.

Below is a list of parallel compiler directives and what they do:

/**********************************************************************/
 /*  Compiler Directives                                                */
 /**********************************************************************/
 
 #if RP_HOST
    /* only host process is involved */
 #endif
 
 #if RP_NODE
    /* only compute nodes are involved */
 #endif
 
 
 /*********************************************************************/
 /*  Negated forms that are more commonly used                             */
 /*********************************************************************/
 
 #if !RP_HOST
    /* only compute nodes are involved */
 #endif
 
 #if !RP_NODE
    /* only host process is involved */
 #endif
 

The following simple UDF shows the use of compiler directives. The adjust function is used to define a function called where_am_i. This function queries to determine which type of process is executing and then displays a message on that computed node’s monitor.

Example

/*****************************************************
  Simple UDF that uses compiler directives
 *****************************************************/
 #include "udf.h"
 DEFINE_ADJUST(where_am_i, domain)
 {
 #if RP_HOST
    Message("I am in the host process\n");
 #endif /* RP_HOST */
 
 #if RP_NODE
    Message("I am in the node process with ID %d\n",myid);
    /* myid is a global variable which is set to the multiport ID for
    each node */
 #endif /* RP_NODE */
 } 

This simple allocation of functionality between the different types of processes is useful in a limited number of practical situations. For example, you may want to display a message on the compute nodes when a particular computation is being run (by using RP_NODE or !RP_HOST). Or, you can also choose to designate the host process to display messages (by using RP_HOST or !RP_NODE). Usually you want messages written only once by the host process. Simple messages such as "Running the Adjust Function" are straightforward. Alternatively, you may want to collect data from all the nodes and print the total once, from the host. To perform this type of operation your UDF will need some form of communication between processes. The most common mode of communication is between the host and the node processes.

There are two sets of similar macros that can be used to send data between the host and the compute nodes: host_to_node_type_num and node_to_host_type_num.

host_to_node_type_num(val_1,val_2,...,val_num); 

/* integer and real variables passed from host to nodes */
 host_to_node_int_1(count);
 host_to_node_real_7(len1, len2, width1, width2, breadth1, breadth2, vol); 

/* string and array variables passed from host to nodes */
 char wall_name[]="wall-17";
 int thread_ids[10] = {1,29,5,32,18,2,55,21,72,14};
 
 host_to_node_string(wall_name,8); /* remember terminating NUL character */
 host_to_node_int(thread_ids,10); 

node_to_host_type_num(val_1,val_2,...,val_num); 

/* integer and real variables passed from compute node-0 to host */
 node_to_host_int_1(count);
 node_to_host_real_7(len1, len2, width1, width2, breadth1, breadth2, vol); 

 /* string and array variables passed from compute node-0 to host */
 char *string;
 int string_length;
 real vel[ND_ND];
 
 node_to_host_string(string,string_length);
 node_to_host_real(vel,ND_ND); 

There are a number of macros available in parallel Ansys Fluent that expand to logical tests. These logical macros, referred to as "predicates", are denoted by the suffix P and can be used as test conditions in your UDF. The following predicates return TRUE if the condition in the parenthesis is met.

/* predicate definitions from para.h header file */
 
 # define MULTIPLE_COMPUTE_NODE_P (compute_node_count > 1)
 # define ONE_COMPUTE_NODE_P (compute_node_count == 1)
 # define ZERO_COMPUTE_NODE_P (compute_node_count == 0) 

There are a number of predicates that allow you to test the identity of the node process in your UDF, using the compute node ID. A compute node’s ID is stored as the global integer variable myid (see Process Identification). Each of the macros listed below tests certain conditions of myid for a process. For example, the predicate I_AM_NODE_ZERO_P compares the value of myid with the compute node-0 ID and returns TRUE when they are the same. I_AM_NODE_SAME_P(n), on the other hand, compares the compute node ID that is passed in n with myid. When the two IDs are the same, the function returns TRUE. Node ID predicates are often used in conditional-if statements in UDFs.

/* predicate definitions from para.h header file */
 
 # define I_AM_NODE_HOST_P (myid == host)
 # define I_AM_NODE_ZERO_P (myid == node_zero)
 # define I_AM_NODE_ONE_P (myid == node_one)
 # define I_AM_NODE_LAST_P (myid == node_last)
 # define I_AM_NODE_SAME_P(n) (myid == (n))
 # define I_AM_NODE_LESS_P(n) (myid < (n))
 # define I_AM_NODE_MORE_P(n) (myid > (n)) 

Recall that from Cells and Faces in a Partitioned Mesh, a face may appear in one or two partitions but in order that summation operations do not count it twice, it is officially allocated to only one of the partitions. The tests above are used with the neighboring cell’s partition ID to determine if it belongs to the current partition. The convention that is used is that the smaller-numbered compute node is assigned as the "principal" compute node for that face. PRINCIPAL_FACE_P returns TRUE if the face is located on its principal compute node. The macro can be used as a test condition when you want to perform a global sum on faces and some of the faces are partition boundary faces. Below is the definition of PRINCIPAL_FACE_P from para.h. See Cells and Faces in a Partitioned Mesh for more information about PRINCIPAL_FACE_P.

/* predicate definitions from para.h header file */
 # define PRINCIPAL_FACE_P(f,t) (!TWO_CELL_FACE_P(f,t) || \
  PRINCIPAL_TWO_CELL_FACE_P(f,t))
 
 # define PRINCIPAL_TWO_CELL_FACE_P(f,t) \
  (!(I_AM_NODE_MORE_P(C_PART(F_C0(f,t),THREAD_T0(t))) || \
  I_AM_NODE_MORE_P(C_PART(F_C1(f,t),THREAD_T1(t))))) 

Global reduction operations are those that collect data from all of the compute nodes, and reduce the data to a single value, or an array of values. These include operations such as global summations, global maximums and minimums, and global logicals. These macros begin with the prefix PRF_G and are defined in prf.h. Global summation macros are identified by the suffix SUM, global maximums by HIGH, and global minimums by LOW. The suffixes AND and OR identify global logicals.

The variable data types for each macro are identified in the macro name, where R denotes real data types, I denotes integers, and L denotes logicals. For example, the macro PRF_GISUM finds the summation of integers over the compute nodes.

Each of the global reduction macros discussed in the following sections has two different versions: one takes a single variable argument, while the other takes a variable array. Macros with a 1 (one) appended to the end of the name take one argument, and return a single variable as the global reduction result. For example, the macro PRF_GIHIGH1(x) expands to a function that takes one argument x and computes the maximum of the variable x among all of the compute nodes, and returns it. The result can then be assigned to another variable (such as y), as shown in the following example.

Example: Global Reduction Variable Macro

{
 int y;
 int x = myid;
 y = PRF_GIHIGH1(x); /* y now contains the same number (compute_node_count
       - 1) on all the nodes */
 } 

Macros without a 1 suffix, on the other hand, compute global reduction variable arrays. These macros take three arguments: x, N, and iwork where x is an array, N is the number of elements in the array, and iwork is an array that is of the same type and size as x which is needed for temporary storage. Macros of this type are passed an array x and the elements of array x are filled with the new result after returning from the function. For example, the macro PRF_GIHIGH(x,N,iwork) expands to a function that computes the maximum of each element of the array x over all the compute nodes, uses the array iwork for temporary storage, and modifies array x by replacing each element with its resulting global maximum. The function does not return a value.

Example: Global Reduction Variable Array Macro

{
 real x[N], iwork[N];
 /* The elements of x are set in the working array here and will
  have different values on each compute node.
  In this case, x[0] could be the maximum cell temperature of all
  the cells on the compute node. x[1] the maximum pressure, x[2]
  the maximum density, etc.
 */
  PRF_GRHIGH(x,N,iwork); /* The maximum value for each value over
        all the compute nodes is found here */
  /* The elements of x on each compute node now hold the same
  maximum values over all the compute nodes for temperature,
  pressure, density, etc. */
 } 

In general, cells and faces have a partition ID that is numbered from 0 to n-1, where n is the number of compute nodes. The partition IDs of cells and faces are stored in the variables C_PART and F_PART, respectively. C_PART(c,tc) stores the integer partition ID of a cell and F_PART(f,tf) stores the integer partition ID of a face.

Note that myid can be used in conjunction with the partition ID, since the partition ID of an exterior cell is the ID of the neighboring compute node.

 

7.3.2.6.1. Cell Partition IDs

For interior cells, the partition ID is the same as the compute node ID. For exterior cells, the compute node ID and the partition ID are different. For example, in a parallel system with two compute nodes (0 and 1), the exterior cells of compute node-0 have a partition ID of 1, and the exterior cells of compute node-1 have a partition ID of 0 (Figure 7.14: Partition Ids for Cells and Faces in a Compute Node).

Figure 7.14: Partition Ids for Cells and Faces in a Compute Node

You can direct Ansys Fluent to display messages on a host or node process using the Message utility. To do this, simply use a conditional if statement and the appropriate compiler directive (such as #if RP_NODE) to select the process(es) you want the message to come from. This is demonstrated in the following example:

Example

#if RP_NODE
  Message("Total Area Before Summing %f\n",total\_area);
 #endif /* RP_NODE */ 

In this example, the message will be sent by the compute nodes. (It will not be sent by the host.)

Message0 is a specialized form of the Message utility. Message0 will send messages from compute node-0 only and is ignored on the other compute nodes, without having to use a compiler directive.

Example

/* Let Compute Node-0 display messages */
 
 Message0("Total volume = %f\n",total_volume); 

High-level communication macros of the form node_to_host... and host_to_node... that are described in Communicating Between the Host and Node Processes are typically used when you want to send data from the host to all of the compute nodes, or from node-0 to the host. You cannot, however, use these high-level macros when you need to pass data between compute nodes, or pass data from all of the compute nodes to compute node-0. In these cases, you can use special message passing macros described in this section.

Note that the higher-level communication macros expand to functions that perform a number of lower-level message passing operations which send sections of data as single arrays from one process to another process. These lower-level message passing macros can be easily identified in the macro name by the characters SEND and RECV. Macros that are used to send data to processes have the prefix PRF_CSEND, whereas macros that are used to receive data from processes have the prefix PRF_CRECV. Data that is to be sent or received can belong to the following data types: character (CHAR), integer (INT), REAL and logical (BOOLEAN). BOOLEAN variables are TRUE or FALSE. REAL variables are assigned as float data types when running a single precision version of Ansys Fluent and double when running double precision. Message passing macros are defined in the prf.h header file and are listed below.

/* message passing macros */
 
 PRF_CSEND_CHAR(to, buffer, nelem, tag)
 PRF_CRECV_CHAR (from, buffer, nelem, tag)
 PRF_CSEND_INT(to, buffer, nelem, tag)
 PRF_CRECV_INT(from, buffer, nelem, tag)
 PRF_CSEND_REAL(to, buffer, nelem, tag)
 PRF_CRECV_REAL(from, buffer, nelem, tag)
 PRF_CSEND_BOOLEAN(to, buffer, nelem, tag)
 PRF_CRECV_BOOLEAN(from, buffer, nelem, tag) 

There are four arguments to the message-passing macros. For ‘send’ messages:

For ‘receive’ messages:

Note that if variables that are to be sent or received are defined in your function as real variables, then you can use the message passing macros with the _REAL suffix. The compiler will then substitute PRF_CSEND_DOUBLE or PRF_CRECV_DOUBLE if you are running double precision and PRF_CSEND_FLOAT or PRF_CRECV_FLOAT, for single precision.

Because message-passing macros are low-level macros, you will need to make sure that when a message is sent from a node process, a corresponding receiving macro appears in the receiving-node process. Note that your UDF cannot directly send messages from a compute node (other than 0) to the host using message-passing macros. They can send messages indirectly to the host through compute node-0. For example, if you want your parallel UDF to send data from all of the compute nodes to the host for postprocessing purposes, the data will first have to be passed from each compute node to compute node-0, and then from compute node-0 to the host. In the case where the compute node processes send a message to compute node-0, compute node-0 must have a loop to receive the N messages from the N nodes.

Below is an example of a compiled parallel UDF that utilizes message passing macros PRF_CSEND and PRF_CRECV. Refer to the comments (*/) in the code, for details about the function.

Example: Message Passing

#include "udf.h"
 #define WALLID 3
 
 DEFINE_ON_DEMAND(face_p_list)
 {
   #if !RP_HOST /* Host will do nothing in this udf. */
      face_t f;
      Thread *tf;
      Domain *domain;
      real *p_array;
      real x[ND_ND], (*x_array)[ND_ND];
      int n_faces, i, j;

      domain=Get_Domain(1); /* Each Node will be able to access
             its part of the domain */

      tf=Lookup_Thread(domain, WALLID); /* Get the thread from the domain */

      /* The number of faces of the thread on nodes 1,2... needs to be sent
      to compute node-0 so it knows the size of the arrays to receive
      from each */

      n_faces=THREAD_N_ELEMENTS_INT(tf);

      /* No need to check for Principal Faces as this UDF
      will be used for boundary zones only */
 
    if(! I_AM_NODE_ZERO_P) /* Nodes 1,2... send the number of faces */
      {
       PRF_CSEND_INT(node_zero, &n_faces, 1, myid);
      }

   /* Allocating memory for arrays on each node */  
   p_array=(real *)malloc(n_faces*sizeof(real));
    x_array=(real (*)[ND_ND])malloc(ND_ND*n_faces*sizeof(real));

    begin_f_loop(f, tf)
       /* Loop over interior faces in the thread, filling p_array
       with face pressure and x_array with centroid  */
      {
         p_array[f] = F_P(f, tf);
         F_CENTROID(x_array[f], f, tf);
      }
    end_f_loop(f, tf)
 
 /* Send data from node 1,2, ... to node 0 */
 Message0("\nstart\n");

    if(! I_AM_NODE_ZERO_P) /* Only SEND data from nodes 1,2... */
    {
       PRF_CSEND_REAL(node_zero, p_array, n_faces, myid);
       PRF_CSEND_REAL(node_zero, x_array[0], ND_ND*n_faces, myid);
    }
  else
 
    {/* Node-0 has its own data,
      so list it out first */
     Message0("\n\nList of Pressures...\n");
     for(j=0; j<n_faces; j++)
        /* n_faces is currently node-0 value */
       {
 # if RP_3D
         Message0("%12.4e %12.4e %12.4e %12.4e\n",
         x_array[j][0], x_array[j][1], x_array[j][2], p_array[j]);
 # else /* 2D */
         Message0("%12.4e %12.4e %12.4e\n",
         x_array[j][0], x_array[j][1], p_array[j]);
 # endif
     }
  }
 
 /* Node-0 must now RECV data from the other nodes and list that too */
    if(I_AM_NODE_ZERO_P)
    {
      compute_node_loop_not_zero(i)
       /* See para.h for definition of this loop */
      {
         PRF_CRECV_INT(i, &n_faces, 1, i);
          /* n_faces now value for node-i */
         /* Reallocate memory for arrays for node-i */
               p_array=(real *)realloc(p_array, n_faces*sizeof(real));
         x_array=(real(*)[ND_ND])realloc(x_array,ND_ND*n_faces*sizeof(real));

         /* Receive data */
         PRF_CRECV_REAL(i, p_array, n_faces, i);
         PRF_CRECV_REAL(i, x_array[0], ND_ND*n_faces, i);
         for(j=0; j<n_faces; j++)
           {
 # if RP_3D
              Message0("%12.4e %12.4e %12.4e %12.4e\n",
              x_array[j][0], x_array[j][1], x_array[j][2], p_array[j]);
 # else /* 2D */
              Message0("%12.4e %12.4e %12.4e\n",
              x_array[j][0], x_array[j][1], p_array[j]);
 # endif
           }
       }
    }

    free(p_array); /* Each array has to be freed before function exit */
    free(x_array);
 
 #endif /* ! RP_HOST */
 } 

EXCHANGE_SVAR_MESSAGE, EXCHANGE_SVAR_MESSAGE_EXT, and EXCHANGE_SVAR_FACE_MESSAGE can be used to exchange storage variables (SV_...) between compute nodes. EXCHANGE_SVAR_MESSAGE and EXCHANGE_SVAR_MESSAGE_EXT exchange cell data between compute nodes, while EXCHANGE_SVAR_FACE_MESSAGE exchanges face data. EXCHANGE_SVAR_MESSAGE is used to exchange data over regular exterior cells, while EXCHANGE_SVAR_MESSAGE_EXT is used to exchange data over regular and extended exterior cells. Note that compute nodes are ‘virtually’ synchronized when an EXCHANGE macro is used; receiving compute nodes wait for data to be sent, before continuing.

/* Compute Node Exchange Macros  */
 
 EXCHANGE_SVAR_FACE_MESSAGE(domain, (SV_P, SV_NULL));
 EXCHANGE_SVAR_MESSAGE(domain, (SV_P, SV_NULL)); 
 EXCHANGE_SVAR_MESSAGE_EXT(domain, (SV_P, SV_NULL));

EXCHANGE_SVAR_FACE_MESSAGE() is rarely needed in UDFs. You can exchange multiple storage variables between compute nodes. Storage variable names are separated by commas in the argument list and the list is ended by SV_NULL. For example, EXCHANGE_SVAR_MESSAGE(domain, (SV_P, SV_T, SV_NULL)) is used to exchange cell pressure and temperature variables. You can determine a storage variable name from the header file that contains the variable’s definition statement. For example, suppose you want to exchange the cell pressure (C_P) with an adjacent compute node. You can look at the header file that contains the definition of C_P (mem.h) and determine that the storage variable for cell pressure is SV_P. You will need to pass the storage variable to the exchange macro.