Differences

This shows you the differences between two versions of the page.

--- s800_daq_tools [2016/02/29 11:50]
pereira [Readout GUI]
+++ s800_daq_tools [2016/02/29 14:59]
pereira [Old processes still running]
@@ Line 2: / Line 2: @@
   * [[#Running the S800 DAQ]]
-    * [[#Run Control Window]]
+    * [[#Readout GUI]]
-    * [[#EventBuilder Window]]
     * [[#Mesytec CFD GUI]]
     * [[#Delay Window]]
     * [[#Trigger GUI]]
     * [[#Scaler Display]]
-    * [[#Troubleshooting]]
+  * [[#Troubleshooting]]
-  * [[#S800 DAQ status tool]]
-    * [[#Readout GUI tab]]
-    * [[#Eventbuilder tab]]
-    * [[#Controls tab]]
   * [[#How to run combined S800 + GRETINA]]
@@ Line 92: / Line 87: @@
-==== Troubleshooting ====
+===== Troubleshooting =====
-In the rare case that the RunControl GUI gets "frozen", it is possible to reboot the whole system manually. This operation requires to turn off the VME and CAMAC crates, and the spdaq computer, all them located in the rack seating near the FP box in the S3 vault. Then, the system must be turned back on in the following order. First, CAMAC crate, second VME crate, and third spdaq.
+There are several common reasons why the S800 DAQ may fails to start. The list below describes what to do to fix these problems.
-===== S800 DAQ status tool =====
+  * [[#No data sources defined]]
-In order to remotely monitor the status of the S800 DAQ, a tool has been created which can show statistics and current information about the present running instances of the Readout GUI and eventbuilder programs. This is accessed by the following:
+  * [[#CCUSB and/or VMUSB claimed by existing programs]]
+  * [[#Manual rebooting]]
-. On spdaq48 do:
-**/user/s800/operations/daq/usb/s800status/comms**
-which will open the status GUI.
+==== Old processes still running ====
+Sometimes, particularly if the previous S800 DAQ session ended in an uncontrolled way, there may be old processes running (CCUSB and/or VMUSB and/or S800 event builder) that will prevent Readout GUI to begin a run. Whenever this happens (typically after clicking ''Begin''), the system sends an error message complaining about one of these processes still running. If you find yourself in this situation try:
+      - Make sure that you exit your Readout GUI properly
+      - Type ''ps aux | grep Readout'' to ensure that Readout is indeed not running
+      - Type ''/usr/opt/nscldaq/xxxx/bin/ringbuffer status'', where xxxx is the daq version, e.g. 11.0-020
+      - You will see a list of ringbuffers with information about their status (see figure below). Check that the producer value of each ringbuffer (see column ''producer'') is **-1**. (Don't worry about the ringbuffer **s800filter**, which is automatically killed when starting ReadoutGUI)
+      - If the producer value of a ringbuffer is not -1, take note of the PID, and kill the process by typing ''kill -9 PID''
+      - You should now be able to run Readout GUI and begin a run without problem
-There are tabs for connections to the Readout GUI, the eventbuilder and a controls tab.
+{{:wiki:ringbuffer1.png?650|Ringbuffer status}}
-==== Readout GUI tab ====
+==== No data sources defined ====
-The communications port to the Readout GUI is tcp:7999 and it runs by default on spdaq48. These options are selected by default in the server and port drop down boxes, but other values can be entered here by hand. Clicking "Get Status" will connect to this server/port combination and attempt to fill in the Run Info section with the returned information. The status will also be updated.
+When clicking ''Begin'' in the S800 Readout GUI, you get the error message: "No data sources are running so a run cannot be started". Very likely, the setting file **.settings.tcl** is missing. On a Linux session in u6pc5, type ''ls -lisa ~/stagearea''. You should see a hidden file **.settings.tcl**. If no, try the following:
+      - Go to **/mnt/daqtesting/s800/exxxxlast**, where **exxxxlast** corresponds to the experiment number of the last (successfully) run experiment
+      - Copy the file **.settings.tcl** from that directory to the current stagearea. **NOTE**: we are assuming that the stagearea is pointing to the new experiment directory **/mnt/daqtesting/s800/exxxxnew**
+      - If you cannot find that file, there is an old version that can be copied from the directory **/user/s800/converged_daq**
+      - You should now be able to run ReadoutGUI without problem
+==== CCUSB and/or VMUSB claimed by existing programs ====
+One of the most common reasons why the S800 DAQ fails to start is because the CCUSB and/or VMUSB are claimed by existing programs. This is shown by the error messages displayed by the system. If you find yourself in this situation try the following:
+      - Log on to S800 spdaq (spdaq19 as of October 2015)
+      - Type ''ps aux | grep Readout''
+      - Use ''kill -9 PID'', where PID identifies any CCUSBReadout or VMUSBReadout processes that show up
+      - Use ''kill -9 PID'', where PID identifies any tclsh ReadoutShell process that might show up
+      - Try ''godaq'' to see if this works now
+      - If this didn't help try rebooting the S800 DAQ manually (see below)
+==== Manual rebooting ====
+In the rare case that the RunControl GUI gets "frozen", it is possible to reboot the whole system manually. This operation requires to turn off the VME and CAMAC crates, and the spdaq computer, all them located in the rack seating near the FP box in the S3 vault. Then, the system must be turned back on in the following order. First, CAMAC crate, second VME crate, and third spdaq.
-{{:wiki:comms.png?500|S800 Status GUI}}
-==== Eventbuilder tab ====
+===== How to run combined S800 + GRETINA =====
-The communications port to the eventbuilder is tcp:8100 and it runs by default on spdaq48. These options can be changed in a manner identical to the Readout GUI tab. Information about the current statistics (input/output buffers, etc) will be updated, as will the connections information. In this scroll box any connection to which the eventbuilder must write data is listed in the form: remote host:local port.
+OBSOLETE. Documentation is currently under development.
-{{:wiki:evbuild.png?500|Eventbuilder tab}}
-This is the only information available to the tcp protocol, but the pid of the processes responsible for these connections can be investigated by using the following commands:
-. For each remote host, log in as each user which may be connected to the DAQ (ie e11007, s800). Unfortunately, root access is required to see this information for every user, so seperate logins must be made for each possible user.
-. Do **lsof -i | grep port** (where port is replaced by the local port information obtained from the status gui. This will look similar to:
-<code>
-netcat.tc 2498 e11007   10u  IPv4  11103      0t0  TCP spdaq44.nscl.msu.edu:58207->spdaq48.nscl.msu.edu:9002 (ESTABLISHED)
-</code>
-where the second column shows the PID of the connecting process. In this case the pid is 2498.
-. Do **ps -aux | grep PID**. This will look similar to:
-<code>
-e11007    2498  0.0  0.0  12052  2252 ?        Sl   12:31   0:20 /usr/bin/tclsh /soft/intel/lucid/daq/10.1/TclLibs/s800/../../bin/netcat.tcl spdaq48 9002
-</code>
-which gives the program name associated with the PID that is connecting to the eventbuilder.
-If the eventbuilder is hanging, which is evidenced by non-increasing statistics and no data/triggers, then it is possible that one of these processes has crashed in a manner that holds open the tcp connection to the eventbuilder. The process can be killed, which will return the DAQ to a normal running mode.
-==== Controls tab ====
-This tab allows forced control of the Readout GUI, and is for advanced users only. In the event that the run must be stopped, or the DAQ software exited without access to the Readout GUI itself these buttons may be used to perform these actions. The run MUST be ended before using the Force Exit button, or it is likely that the USB crate controllers will become stuck in DAQ mode and the crates will have to be reset.
-{{:wiki:controls.png?500|Controls tab}}
-===== How to run combined S800 + GRETINA =====
 In order to run the combined S800 + Gretina daq, the following steps need to be taken:

S800 Documentation

User Tools

Site Tools

Differences

Page Tools