| Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
s800_daq_tools [2016/02/29 11:51]
pereira [S800 DAQ status tool] |
s800_daq_tools [2016/03/16 15:58]
pereira [S800 Software tools] |
| |
| * [[#Running the S800 DAQ]] | * [[#Running the S800 DAQ]] |
| * [[#Run Control Window]] | * [[#Readout GUI]] |
| * [[#EventBuilder Window]] | |
| * [[#Mesytec CFD GUI]] | * [[#Mesytec CFD GUI]] |
| * [[#Delay Window]] | * [[#Delay Window]] |
| * [[#Trigger GUI]] | * [[#Trigger GUI]] |
| * [[#Scaler Display]] | * [[#Scaler Display]] |
| * [[#Troubleshooting]] | * [[#Troubleshooting]] |
| * [[#S800 DAQ status tool]] | |
| * [[#Readout GUI tab]] | |
| * [[#Eventbuilder tab]] | |
| * [[#Controls tab]] | |
| * [[#How to run combined S800 + GRETINA]] | |
| |
| |
| |
| ==== Readout GUI ==== | ==== Readout GUI ==== |
| The S800 Readout GUI is invoked by either clicking the icon **S800 DAQ** in the desktop of [[Software#u6pc5 (data U6)|u6pc5]] or by navigating in a Linux terminal to the directory **/user/s800/converged_daq** and typing ''godaq''. The Readout GUI window will appear. **IMPORTANT NOTE:** When running Readout GUI, make sure that the environment variables needed by the system are properly defined. This can be done from a Linux terminal by typing: ''. /usr/opt/nscldaq/xxxx/daqsetup.bash'', where ''xxxx'' is the nscldaq version (xxx = 11.0-020, on Feb 2016). | The S800 Readout GUI is invoked by either clicking the icon **S800 DAQ** in the desktop of [[Software#u6pc5 (data U6)|u6pc5]] or by navigating in a Linux terminal to the directory **/user/s800/converged_daq** and typing ''godaq''. The Readout GUI window will appear with three tags labeled "main", "SSHPipe@localhost:0", and "SSHPipe@localhost:1". The first one provides general information about the ReadoutGUI. The second and third tabs provide information about the data sources (from the CCUSB and VMUSB controllers). **IMPORTANT NOTE:** When running Readout GUI, make sure that the environment variables needed by the system are properly defined. This can be done from a Linux session connected to spdaq19 by typing: ''. /usr/opt/nscldaq/xxxx/daqsetup.bash'', where ''xxxx'' is the nscldaq version (xxx = 11.0-020, on Feb 2016). |
| |
| Before beginning taking data is necessary to initialize the system. This is done by clicking ''Start'' in the ReadoutGUI. After initializing the contollers, the ReadoutGUI window will show the ''Begin'' button active. In addition, three additional buttons should appear at the end of the window. They can be used to start the GUI associated with the [[#Mesytec CFD GUI|MCFD]], [[#Delay Window|Delay XLM]], and [[#Trigger GUI|Trigger ULM]] modules. Make sure that the system is fully initializing before opening those GUIs. | Before beginning taking data is necessary to initialize the system. This is done by clicking ''Start'' in the ReadoutGUI. You can inspect the status of each source during their initialization by clicking in one of the tabs "SSHPipe@localhost:0" or "SSHPipe@localhost:1". You will see a series of messages about the different initialization steps. The last message should be "Done". During the initialization process, a fourth tab labeled "ActionFilter" may appear with information from the S800 filter. Sometimes, you may see a warning message about an old "still-running" S800 filter session being killed. That's ok. |
| |
| The figure below shows the Readout GUI window after initializing the system. After clicking ''Begin'', the "Event Builder" window will pop out, displaying information about the Readout session. Data can be recorded on disk by checking out the box ''Record''. To end a data run, simply click ''End''. | |
| | After initializing the contollers, the ReadoutGUI window will show the ''Begin'' button active. In addition, three additional buttons should appear at the end of the window. They can be used to start the GUI associated with the [[#Mesytec CFD GUI|MCFD]], [[#Delay Window|Delay XLM]], and [[#Trigger GUI|Trigger ULM]] modules. Make sure that the system is completely initializing before opening those GUIs. |
| | |
| | The figure below shows the Readout GUI window after initializing the system. After clicking ''Begin'', the "Event Builder" window will pop out, displaying information about the Readout session. (Note that the tab "ActionFilter" may appear if it didn't show up before, when you clicked ''Start''.) Data can be recorded on disk by checking out the box ''Record''. To end a data run, simply click ''End''. |
| |
| |
| The S800 electronics includes a Mesytec CFD (MCFD), used to "filter" the detector signals going to the Scaler and Mesytec TDC. The configuration parameters of the CFD (thresholds, delays, fraction, etc.) can be remotely adjusted via a MCFD GUI developed by the NSCL DAQ group. During tuning of the S800, one typically needs to adjust thresholds only. | The S800 electronics includes a Mesytec CFD (MCFD), used to "filter" the detector signals going to the Scaler and Mesytec TDC. The configuration parameters of the CFD (thresholds, delays, fraction, etc.) can be remotely adjusted via a MCFD GUI developed by the NSCL DAQ group. During tuning of the S800, one typically needs to adjust thresholds only. |
| |
| The MCFD GUI is invoked by either clicking the icon **MCFD GUI** in the desktop of [[Software#u6pc5 (data U6)|u6pc5]] or by navigating in a Linux terminal and typing ''$DAQBIN/MCFDControl16 --protocol usb --serialfile /dev/ttyUSB0''. The environment variable DAQBIN is defined by sourcing the **daqsetup.bash** file in directory **/usr/opt/nscldaq/xxxx/bin**, where xxxx is the nscldaq version number (11.0-015 on Oct 26, 2015). | The MCFD GUI can be started in three different ways: 1) by clicking the button ''Launch MCFD GUI'' in [[#Readout GUI|ReadoutGUI]]; 2) by clicking the icon **MCFD GUI** in the desktop of [[Software#u6pc5 (data U6)|u6pc5]]; 3) by navigating in a Linux terminal and typing ''$DAQBIN/MCFDControl16 --protocol usb --serialfile /dev/ttyUSB0''. The environment variable DAQBIN is defined by sourcing the **daqsetup.bash** file in directory **/usr/opt/nscldaq/xxxx/bin**, where xxxx is the nscldaq version number (11.0-015 on Oct 26, 2015). |
| |
| |
| |
| |
| ==== Troubleshooting ==== | ===== Troubleshooting ===== |
| In the rare case that the RunControl GUI gets "frozen", it is possible to reboot the whole system manually. This operation requires to turn off the VME and CAMAC crates, and the spdaq computer, all them located in the rack seating near the FP box in the S3 vault. Then, the system must be turned back on in the following order. First, CAMAC crate, second VME crate, and third spdaq. | There are several common reasons why the S800 DAQ may fails to start. The list below describes what to do to fix these problems. |
| | * [[#Old processes still running]] |
| | * [[#No data sources defined]] |
| | * [[#CCUSB and/or VMUSB claimed by existing programs]] |
| | * [[#Manual rebooting]] |
| |
| ==== Readout GUI tab ==== | |
| The communications port to the Readout GUI is tcp:7999 and it runs by default on spdaq48. These options are selected by default in the server and port drop down boxes, but other values can be entered here by hand. Clicking "Get Status" will connect to this server/port combination and attempt to fill in the Run Info section with the returned information. The status will also be updated. | |
| |
| {{:wiki:comms.png?500|S800 Status GUI}} | |
| |
| | ==== Old processes still running ==== |
| | Sometimes, particularly if the previous S800 DAQ session ended in an uncontrolled way, there may be old processes running (CCUSB and/or VMUSB and/or S800 event builder) that will prevent Readout GUI to begin a run. Whenever this happens (typically after clicking ''Begin'' in Readout GUI), the system sends an error message complaining about one of these processes still running. If you find yourself in this situation try: |
| | - Exit your Readout GUI properly |
| | - On a Linux session, connect to spdaq19 by typing ''ssh -Y s800@spdaq19'' (the S800 password can be found in the S800 binder that is kept in the data-U1 logbook cabinet) |
| | - Type ''ps aux | grep Readout'' to ensure that Readout is indeed not running |
| | - Type ''/usr/opt/nscldaq/xxxx/bin/ringbuffer status'', where xxxx is the daq version, e.g. 11.0-020 |
| | - You will see a list of ringbuffers with information about their status (see figure below). Check that the producer value of each ringbuffer (see column ''producer'') is **-1**. (Don't worry about the ringbuffer **s800filter**, which is automatically killed when starting ReadoutGUI) |
| | - If the producer value of a ringbuffer is not -1, take note of the PID, and kill the process by typing ''kill -9 PID'' |
| | - You should now be able to run Readout GUI and begin a run without problem |
| |
| ==== Eventbuilder tab ==== | |
| The communications port to the eventbuilder is tcp:8100 and it runs by default on spdaq48. These options can be changed in a manner identical to the Readout GUI tab. Information about the current statistics (input/output buffers, etc) will be updated, as will the connections information. In this scroll box any connection to which the eventbuilder must write data is listed in the form: remote host:local port. | |
| |
| {{:wiki:evbuild.png?500|Eventbuilder tab}} | {{:wiki:ringbuffer1.png?650|Ringbuffer status}} |
| | |
| This is the only information available to the tcp protocol, but the pid of the processes responsible for these connections can be investigated by using the following commands: | |
| | |
| 1. For each remote host, log in as each user which may be connected to the DAQ (ie e11007, s800). Unfortunately, root access is required to see this information for every user, so seperate logins must be made for each possible user. | |
| | |
| 2. Do **lsof -i | grep port** (where port is replaced by the local port information obtained from the status gui. This will look similar to: | |
| | |
| <code> | |
| netcat.tc 2498 e11007 10u IPv4 11103 0t0 TCP spdaq44.nscl.msu.edu:58207->spdaq48.nscl.msu.edu:9002 (ESTABLISHED) | |
| </code> | |
| | |
| where the second column shows the PID of the connecting process. In this case the pid is 2498. | |
| | |
| 3. Do **ps -aux | grep PID**. This will look similar to: | |
| | |
| | |
| <code> | |
| e11007 2498 0.0 0.0 12052 2252 ? Sl 12:31 0:20 /usr/bin/tclsh /soft/intel/lucid/daq/10.1/TclLibs/s800/../../bin/netcat.tcl spdaq48 9002 | |
| </code> | |
| | |
| which gives the program name associated with the PID that is connecting to the eventbuilder. | |
| | |
| If the eventbuilder is hanging, which is evidenced by non-increasing statistics and no data/triggers, then it is possible that one of these processes has crashed in a manner that holds open the tcp connection to the eventbuilder. The process can be killed, which will return the DAQ to a normal running mode. | |
| | |
| | |
| | |
| ===== How to run combined S800 + GRETINA ===== | |
| | |
| In order to run the combined S800 + Gretina daq, the following steps need to be taken: | |
| | |
| 1. On spdaq48 in /user/s800/operations/daq/usb/TclXXUSB do: | |
| | |
| **wish ./RunControl.tcl** | |
| | |
| or, on devop2 (the S800 MAC machine) | |
| | |
| click on the operations folder on the lower icon bar and select runcontrol.command | |
| | |
| 2. On a dataU machine logged in as the experimental account (e11007 for the June 2012 testing period) do | |
| | |
| **godaq** | |
| | |
| 3. On spdaq44 do: | |
| | |
| **/usr/opt/daq/10.1/bin/ringbuffer status**. The output should look similar to: | |
| | |
| <code> | |
| +----------------+------------+-------+-------------+--------+---------+---------+------+-------------+ | |
| |Name |data-size(k)|free(k)|max_consumers|producer|maxget(k)|minget(k)|client|clientdata(k)| | |
| +----------------+------------+-------+-------------+--------+---------+---------+------+-------------+ | |
| |0400x |8195 |8195 |100 |-1 |0 |0 |- |- | | |
| |s800 |8195 |8195 |100 |7485 |0 |0 |- |- | | |
| |- |- |- |- |- |- |- |9143 |0 | | |
| |segatest |8195 |8195 |100 |-1 |0 |0 |- |- | | |
| |spdaq48.0400x |8195 |8195 |100 |17205 |0 |0 |- |- | | |
| |spdaq48.segatest|8195 |8195 |100 |-1 |0 |0 |- |- | | |
| +----------------+------------+-------+-------------+--------+---------+---------+------+-------------+ | |
| </code> | |
| | |
| Upon start of run there will be another client of the s800 ring, which is the pushToGEB process. This should die on end run. | |
| |
| | ==== No data sources defined ==== |
| | When clicking ''Begin'' in the S800 Readout GUI, you get the error message: "No data sources are running so a run cannot be started". Very likely, the setting file **.settings.tcl** is missing. On a Linux session in u6pc5, type ''ls -lisa ~/stagearea''. You should see a hidden file **.settings.tcl**. If no, try the following: |
| | - Go to **/mnt/daqtesting/s800/exxxxlast**, where **exxxxlast** corresponds to the experiment number of the last (successfully) run experiment |
| | - Copy the file **.settings.tcl** from that directory to the current stagearea. **NOTE**: we are assuming that the stagearea is pointing to the new experiment directory **/mnt/daqtesting/s800/exxxxnew** |
| | - If you cannot find that file, there is an old version that can be copied from the directory **/user/s800/converged_daq** |
| | - You should now be able to run ReadoutGUI without problem |
| | ==== CCUSB and/or VMUSB claimed by existing programs ==== |
| | One of the most common reasons why the S800 DAQ fails to start is because the CCUSB and/or VMUSB are claimed by existing programs. This is shown by the error messages displayed by the system. If you find yourself in this situation try the following: |
| | - Log on to S800 spdaq (spdaq19 as of October 2015) |
| | - Type ''ps aux | grep Readout'' |
| | - Use ''kill -9 PID'', where PID identifies any CCUSBReadout or VMUSBReadout processes that show up |
| | - Use ''kill -9 PID'', where PID identifies any tclsh ReadoutShell process that might show up |
| | - Try ''godaq'' to see if this works now |
| | - If this didn't help try rebooting the S800 DAQ manually (see below) |
| | ==== Manual rebooting ==== |
| | In the rare case that the RunControl GUI gets "frozen", it is possible to reboot the whole system manually. This operation requires to turn off the VME and CAMAC crates, and the spdaq computer, all them located in the rack seating near the FP box in the S3 vault. Then, the system must be turned back on in the following order. First, CAMAC crate, second VME crate, and third spdaq. |
| |
| The S800 daq will now be in slave mode, and run control is done from the NSCL daq window. | |
| |
| |
| |
