Skip to content
# Download the last version on 
# on /eos/user/s/sterbini/MD_ANALYSIS/cl2pdExamplePy3.ipynb
# See the details at https://github.com/sterbini/cl2pd
# download it from  SWAN terminal by
# pip install --user git+https://github.com/sterbini/cl2pd.git

import cl2pd
from cl2pd import importData
pd=importData.pd     #is the pandas package
cals=importData.cals #is the ldb variable of pytimber package
!pwd
/eos/home-s/sterbini/MD_ANALYSIS

From CALS to pandas DataFrame

We use the precious package from R. De Maria et al. (pytimber, https://github.com/rdemaria/pytimber) to wrap its output in pandas DataFrames.

cals2pd function

This is the most important function of the importData class. You can use different timezone, in this example we use Central European Time (local time at CERN).

variables=['LHC.BCTDC.A6R4.B1:BEAM_INTENSITY', 'LHC.BCTDC.A6R4.B2:BEAM_INTENSITY']
startTime = pd.Timestamp('2017-10-01 17:30', tz='CET')
endTime = pd.Timestamp('2017-10-01 17:31', tz='CET')
raw_data = importData.cals2pd(variables,startTime,endTime)
raw_data.head()
LHC.BCTDC.A6R4.B1:BEAM_INTENSITY LHC.BCTDC.A6R4.B2:BEAM_INTENSITY
2017-10-01 15:30:00+00:00 1.470000e+09 270000000.0
2017-10-01 15:30:01+00:00 1.810000e+09 310000000.0
2017-10-01 15:30:02+00:00 1.690000e+09 -310000000.0
2017-10-01 15:30:03+00:00 1.990000e+09 -720000000.0
2017-10-01 15:30:04+00:00 1.270000e+09 220000000.0
raw_data['2017-10-01 15:30:01+00:00':'2017-10-01 15:30:03+00:00']
LHC.BCTDC.A6R4.B1:BEAM_INTENSITY LHC.BCTDC.A6R4.B2:BEAM_INTENSITY
2017-10-01 15:30:01+00:00 1.810000e+09 310000000.0
2017-10-01 15:30:02+00:00 1.690000e+09 -310000000.0
2017-10-01 15:30:03+00:00 1.990000e+09 -720000000.0

By default the index timezone is UTC but, even if not encouraged, you can chance the index time zone.

raw_data.index=raw_data.index.tz_convert('CET') # CERN local time
raw_data.head()
LHC.BCTDC.A6R4.B1:BEAM_INTENSITY LHC.BCTDC.A6R4.B2:BEAM_INTENSITY
2017-10-01 17:30:00+02:00 1.470000e+09 270000000.0
2017-10-01 17:30:01+02:00 1.810000e+09 310000000.0
2017-10-01 17:30:02+02:00 1.690000e+09 -310000000.0
2017-10-01 17:30:03+02:00 1.990000e+09 -720000000.0
2017-10-01 17:30:04+02:00 1.270000e+09 220000000.0

You can split the query in different time windows

raw_data = importData.cals2pd(variables,startTime,endTime,split=4,verbose=True)
Time window: 1
t1 is UTC localized: 2017-10-01 15:30:00+00:00
t2 is UTC localized: 2017-10-01 15:30:15+00:00
No fundamental filter.
Elaborating variable: LHC.BCTDC.A6R4.B1:BEAM_INTENSITY
Elaborating variable: LHC.BCTDC.A6R4.B2:BEAM_INTENSITY
Time window: 2
t1 is UTC localized: 2017-10-01 15:30:15+00:00
t2 is UTC localized: 2017-10-01 15:30:30+00:00
No fundamental filter.
Elaborating variable: LHC.BCTDC.A6R4.B1:BEAM_INTENSITY
Elaborating variable: LHC.BCTDC.A6R4.B2:BEAM_INTENSITY
Time window: 3
t1 is UTC localized: 2017-10-01 15:30:30+00:00
t2 is UTC localized: 2017-10-01 15:30:45+00:00
No fundamental filter.
Elaborating variable: LHC.BCTDC.A6R4.B1:BEAM_INTENSITY
Elaborating variable: LHC.BCTDC.A6R4.B2:BEAM_INTENSITY
Time window: 4
t1 is UTC localized: 2017-10-01 15:30:45+00:00
t2 is UTC localized: 2017-10-01 15:31:00+00:00
No fundamental filter.
Elaborating variable: LHC.BCTDC.A6R4.B1:BEAM_INTENSITY
Elaborating variable: LHC.BCTDC.A6R4.B2:BEAM_INTENSITY

You can filter by fundamentals.

t1 = pd.Timestamp('2017-10-01 17:30', tz='CET')
t2 = pd.Timestamp('2017-10-01 17:31', tz='CET')
raw_data = importData.cals2pd(['CPS.TGM:DEST'],t1,t2,fundamental='%TOF%')
raw_data
CPS.TGM:DEST
2017-10-01 15:30:13.900000095+00:00 NTOF
2017-10-01 15:30:17.500000+00:00 NTOF
2017-10-01 15:30:25.900000095+00:00 NTOF
2017-10-01 15:30:27.100000143+00:00 NTOF
2017-10-01 15:30:30.700000048+00:00 NTOF
2017-10-01 15:30:35.500000+00:00 NTOF
2017-10-01 15:30:49.900000095+00:00 NTOF
2017-10-01 15:30:53.500000+00:00 NTOF

cycleStamp2pd function

This function allow to retrieve information from CALS with a list of cyclestamp. For each element in the list it makes a call to pytimber (so can be slow). It is very useful to make aritmetic with the cyclestamps (e.g., for the CERN Injectors Chain).

startTime=pd.Timestamp('2018-03-27 06:00')
endTime=pd.Timestamp('2018-03-27 06:10')
CPSDF=importData.cals2pd(['CPS.LSA:CYCLE'], startTime, endTime, fundamental='%LHC25%')
importData.cycleStamp2pd(['PSB.LSA:CYCLE'],CPSDF.index[1:]-pd.offsets.Milli(635)).head()
PSB.LSA:CYCLE
2018-03-27 06:00:36.065000057+00:00 LHC25_DB_A_PSB
2018-03-27 06:01:13.265000105+00:00 LHC25_DB_A_PSB
2018-03-27 06:01:46.865000010+00:00 LHC25_DB_A_PSB
2018-03-27 06:01:50.464999914+00:00 LHC25_DB_A_PSB
2018-03-27 06:02:24.065000057+00:00 LHC25_DB_A_PSB

LHCFillsByTime function

One can recover the LHC fills between two timestamps.

t1 = pd.Timestamp('2018-04-24 16:57:54.689000+00:00')
t2 = pd.Timestamp('2018-05-24 16:57:54.689000+00:00')
dfFILL=importData.LHCFillsByTime(t1,t2)
dfFILL
mode startTime endTime duration
6611 FILL 2018-04-24 16:57:54.688999891+00:00 2018-04-25 07:15:18.815999985+00:00 14:17:24.127000
6611 SETUP 2018-04-24 17:16:51.466000080+00:00 2018-04-24 18:56:27.641000032+00:00 01:39:36.174999
6611 INJPROT 2018-04-24 18:56:27.641999960+00:00 2018-04-24 19:13:20.552999973+00:00 00:16:52.911000
6611 INJPHYS 2018-04-24 19:13:20.553999901+00:00 2018-04-24 19:52:14.631000042+00:00 00:38:54.077000
6611 PRERAMP 2018-04-24 19:52:14.631999969+00:00 2018-04-24 19:54:30.025000095+00:00 00:02:15.393000
6611 RAMP 2018-04-24 19:54:30.026000023+00:00 2018-04-24 20:14:44.450000048+00:00 00:20:14.424000
6611 FLATTOP 2018-04-24 20:14:44.450999975+00:00 2018-04-24 20:17:40.210000038+00:00 00:02:55.759000
6611 SQUEEZE 2018-04-24 20:17:40.210999966+00:00 2018-04-24 20:28:38.168999910+00:00 00:10:57.957999
6611 ADJUST 2018-04-24 20:28:38.170000076+00:00 2018-04-24 20:35:27.440000057+00:00 00:06:49.269999
6611 STABLE 2018-04-24 20:35:27.440999985+00:00 2018-04-25 06:00:49.973999977+00:00 09:25:22.532999
6611 ADJUST 2018-04-25 06:00:49.974999905+00:00 2018-04-25 07:09:16.572999954+00:00 01:08:26.598000
6611 BEAMDUMP 2018-04-25 07:09:16.573999882+00:00 2018-04-25 07:10:50.769000053+00:00 00:01:34.195000
6611 RAMPDOWN 2018-04-25 07:10:50.769999981+00:00 2018-04-25 07:15:18.815999985+00:00 00:04:28.046000
6612 FILL 2018-04-25 07:15:18.816999912+00:00 2018-04-25 20:58:43.336999893+00:00 13:43:24.519999
6612 SETUP 2018-04-25 07:47:43.690000057+00:00 2018-04-25 08:24:19.234999895+00:00 00:36:35.544999
6612 INJPROT 2018-04-25 08:24:19.236000061+00:00 2018-04-25 08:55:44.729000092+00:00 00:31:25.493000
6612 SETUP 2018-04-25 08:55:44.730000019+00:00 2018-04-25 09:00:50.864000082+00:00 00:05:06.134000
6612 SETUP 2018-04-25 09:00:50.865000010+00:00 2018-04-25 09:06:37.197999954+00:00 00:05:46.332999
6612 INJPROT 2018-04-25 09:06:37.198999882+00:00 2018-04-25 10:17:40.709000111+00:00 01:11:03.510000
6612 INJPHYS 2018-04-25 10:17:40.710000038+00:00 2018-04-25 11:24:47.825999975+00:00 01:07:07.115999
6612 PRERAMP 2018-04-25 11:24:47.826999903+00:00 2018-04-25 11:29:46.378000021+00:00 00:04:58.551000
6612 RAMP 2018-04-25 11:29:46.378999949+00:00 2018-04-25 11:50:12.374000072+00:00 00:20:25.995000
6612 FLATTOP 2018-04-25 11:50:12.375000+00:00 2018-04-25 11:57:10.351999998+00:00 00:06:57.976999
6612 SQUEEZE 2018-04-25 11:57:10.352999926+00:00 2018-04-25 12:08:05.000999928+00:00 00:10:54.648000
6612 ADJUST 2018-04-25 12:08:05.002000093+00:00 2018-04-25 12:16:59.278000116+00:00 00:08:54.276000
6612 STABLE 2018-04-25 12:16:59.279000044+00:00 2018-04-25 20:53:16.812999964+00:00 08:36:17.533999
6612 BEAMDUMP 2018-04-25 20:53:16.813999891+00:00 2018-04-25 20:54:39.635999918+00:00 00:01:22.822000
6612 RAMPDOWN 2018-04-25 20:54:39.637000084+00:00 2018-04-25 20:58:43.336999893+00:00 00:04:03.699999
6613 FILL 2018-04-25 20:58:43.338000059+00:00 2018-04-25 23:57:48.444999933+00:00 02:59:05.106999
6613 SETUP 2018-04-25 22:00:32.696000099+00:00 2018-04-25 22:09:09.723000050+00:00 00:08:37.026999
... ... ... ... ...
6712 SQUEEZE 2018-05-22 16:23:08.066999911+00:00 2018-05-22 16:34:02.688999891+00:00 00:10:54.621999
6712 ADJUST 2018-05-22 16:34:02.690000057+00:00 2018-05-22 16:41:52.858999968+00:00 00:07:50.168999
6712 STABLE 2018-05-22 16:41:52.859999895+00:00 2018-05-23 03:07:48.640000105+00:00 10:25:55.780000
6712 BEAMDUMP 2018-05-23 03:07:48.641000032+00:00 2018-05-23 03:08:02.177000046+00:00 00:00:13.536000
6712 RAMPDOWN 2018-05-23 03:08:02.177999973+00:00 2018-05-23 03:12:09.263999939+00:00 00:04:07.085999
6713 FILL 2018-05-23 03:12:09.265000105+00:00 2018-05-23 04:27:56.147000074+00:00 01:15:46.881999
6713 SETUP 2018-05-23 03:27:44.332000017+00:00 2018-05-23 03:47:09.704999924+00:00 00:19:25.372999
6713 INJPROT 2018-05-23 03:47:09.706000090+00:00 2018-05-23 03:57:04.144000053+00:00 00:09:54.437999
6713 INJPHYS 2018-05-23 03:57:04.144999981+00:00 2018-05-23 04:27:56.147000074+00:00 00:30:52.002000
6714 FILL 2018-05-23 04:27:56.148000002+00:00 2018-05-23 19:52:48.005000114+00:00 15:24:51.857000
6714 INJPROT 2018-05-23 04:28:16.176000118+00:00 2018-05-23 04:31:35.838000059+00:00 00:03:19.661999
6714 INJPHYS 2018-05-23 04:31:35.838999987+00:00 2018-05-23 05:00:19.354000092+00:00 00:28:43.515000
6714 PRERAMP 2018-05-23 05:00:19.355000019+00:00 2018-05-23 05:03:33.286999941+00:00 00:03:13.931999
6714 RAMP 2018-05-23 05:03:33.288000107+00:00 2018-05-23 05:23:47.737999916+00:00 00:20:14.449999
6714 FLATTOP 2018-05-23 05:23:47.739000082+00:00 2018-05-23 05:27:13.437000036+00:00 00:03:25.697999
6714 SQUEEZE 2018-05-23 05:27:13.437999964+00:00 2018-05-23 05:38:08.301000118+00:00 00:10:54.863000
6714 ADJUST 2018-05-23 05:38:08.302000046+00:00 2018-05-23 05:49:04.497999907+00:00 00:10:56.195999
6714 STABLE 2018-05-23 05:49:04.499000072+00:00 2018-05-23 19:48:11.851999998+00:00 13:59:07.352999
6714 BEAMDUMP 2018-05-23 19:48:11.852999926+00:00 2018-05-23 19:48:40.487999916+00:00 00:00:28.634999
6714 RAMPDOWN 2018-05-23 19:48:40.489000082+00:00 2018-05-23 19:52:48.005000114+00:00 00:04:07.516000
6715 FILL 2018-05-23 19:52:48.006000042+00:00 2018-05-23 22:47:48.658999920+00:00 02:55:00.652999
6715 SETUP 2018-05-23 20:23:27.148999929+00:00 2018-05-23 21:28:07.967999935+00:00 01:04:40.819000
6715 INJPROT 2018-05-23 21:28:07.969000101+00:00 2018-05-23 22:20:01.910000086+00:00 00:51:53.940999
6715 INJPHYS 2018-05-23 22:20:01.911000013+00:00 2018-05-23 22:47:48.658999920+00:00 00:27:46.747999
6716 FILL 2018-05-23 22:47:48.660000086+00:00 2018-05-24 04:15:31.756000042+00:00 05:27:43.095999
6716 INJPROT 2018-05-23 22:48:01.979000092+00:00 2018-05-24 02:11:03.315000057+00:00 03:23:01.335999
6716 NOBEAM 2018-05-24 02:11:03.315999985+00:00 2018-05-24 04:15:29.599999905+00:00 02:04:26.283999
6716 CYCLING 2018-05-24 04:15:29.601000071+00:00 2018-05-24 04:15:31.756000042+00:00 00:00:02.154999
6717 FILL 2018-05-24 04:15:31.756999969+00:00 2018-05-24 18:49:18.076999903+00:00 14:33:46.319999
6717 NOBEAM 2018-05-24 05:37:40.792000055+00:00 2018-05-24 18:49:18.076999903+00:00 13:11:37.284999

925 rows × 4 columns

On can change the time zone (in case the user want to do it...)

def _CETconvertMe(x):
    '''
    Return the tz-aware datetime. In case of error returns x.
    '''
    try:
        return x.tz_convert('CET')
    except:
        return x 

dfFILL['startTime']=dfFILL['startTime'].apply(_CETconvertMe)
dfFILL['endTime']=dfFILL['endTime'].apply(_CETconvertMe)
dfFILL
mode startTime endTime duration
6611 FILL 2018-04-24 18:57:54.688999891+02:00 2018-04-25 09:15:18.815999985+02:00 14:17:24.127000
6611 SETUP 2018-04-24 19:16:51.466000080+02:00 2018-04-24 20:56:27.641000032+02:00 01:39:36.174999
6611 INJPROT 2018-04-24 20:56:27.641999960+02:00 2018-04-24 21:13:20.552999973+02:00 00:16:52.911000
6611 INJPHYS 2018-04-24 21:13:20.553999901+02:00 2018-04-24 21:52:14.631000042+02:00 00:38:54.077000
6611 PRERAMP 2018-04-24 21:52:14.631999969+02:00 2018-04-24 21:54:30.025000095+02:00 00:02:15.393000
6611 RAMP 2018-04-24 21:54:30.026000023+02:00 2018-04-24 22:14:44.450000048+02:00 00:20:14.424000
6611 FLATTOP 2018-04-24 22:14:44.450999975+02:00 2018-04-24 22:17:40.210000038+02:00 00:02:55.759000
6611 SQUEEZE 2018-04-24 22:17:40.210999966+02:00 2018-04-24 22:28:38.168999910+02:00 00:10:57.957999
6611 ADJUST 2018-04-24 22:28:38.170000076+02:00 2018-04-24 22:35:27.440000057+02:00 00:06:49.269999
6611 STABLE 2018-04-24 22:35:27.440999985+02:00 2018-04-25 08:00:49.973999977+02:00 09:25:22.532999
6611 ADJUST 2018-04-25 08:00:49.974999905+02:00 2018-04-25 09:09:16.572999954+02:00 01:08:26.598000
6611 BEAMDUMP 2018-04-25 09:09:16.573999882+02:00 2018-04-25 09:10:50.769000053+02:00 00:01:34.195000
6611 RAMPDOWN 2018-04-25 09:10:50.769999981+02:00 2018-04-25 09:15:18.815999985+02:00 00:04:28.046000
6612 FILL 2018-04-25 09:15:18.816999912+02:00 2018-04-25 22:58:43.336999893+02:00 13:43:24.519999
6612 SETUP 2018-04-25 09:47:43.690000057+02:00 2018-04-25 10:24:19.234999895+02:00 00:36:35.544999
6612 INJPROT 2018-04-25 10:24:19.236000061+02:00 2018-04-25 10:55:44.729000092+02:00 00:31:25.493000
6612 SETUP 2018-04-25 10:55:44.730000019+02:00 2018-04-25 11:00:50.864000082+02:00 00:05:06.134000
6612 SETUP 2018-04-25 11:00:50.865000010+02:00 2018-04-25 11:06:37.197999954+02:00 00:05:46.332999
6612 INJPROT 2018-04-25 11:06:37.198999882+02:00 2018-04-25 12:17:40.709000111+02:00 01:11:03.510000
6612 INJPHYS 2018-04-25 12:17:40.710000038+02:00 2018-04-25 13:24:47.825999975+02:00 01:07:07.115999
6612 PRERAMP 2018-04-25 13:24:47.826999903+02:00 2018-04-25 13:29:46.378000021+02:00 00:04:58.551000
6612 RAMP 2018-04-25 13:29:46.378999949+02:00 2018-04-25 13:50:12.374000072+02:00 00:20:25.995000
6612 FLATTOP 2018-04-25 13:50:12.375000+02:00 2018-04-25 13:57:10.351999998+02:00 00:06:57.976999
6612 SQUEEZE 2018-04-25 13:57:10.352999926+02:00 2018-04-25 14:08:05.000999928+02:00 00:10:54.648000
6612 ADJUST 2018-04-25 14:08:05.002000093+02:00 2018-04-25 14:16:59.278000116+02:00 00:08:54.276000
6612 STABLE 2018-04-25 14:16:59.279000044+02:00 2018-04-25 22:53:16.812999964+02:00 08:36:17.533999
6612 BEAMDUMP 2018-04-25 22:53:16.813999891+02:00 2018-04-25 22:54:39.635999918+02:00 00:01:22.822000
6612 RAMPDOWN 2018-04-25 22:54:39.637000084+02:00 2018-04-25 22:58:43.336999893+02:00 00:04:03.699999
6613 FILL 2018-04-25 22:58:43.338000059+02:00 2018-04-26 01:57:48.444999933+02:00 02:59:05.106999
6613 SETUP 2018-04-26 00:00:32.696000099+02:00 2018-04-26 00:09:09.723000050+02:00 00:08:37.026999
... ... ... ... ...
6712 SQUEEZE 2018-05-22 18:23:08.066999911+02:00 2018-05-22 18:34:02.688999891+02:00 00:10:54.621999
6712 ADJUST 2018-05-22 18:34:02.690000057+02:00 2018-05-22 18:41:52.858999968+02:00 00:07:50.168999
6712 STABLE 2018-05-22 18:41:52.859999895+02:00 2018-05-23 05:07:48.640000105+02:00 10:25:55.780000
6712 BEAMDUMP 2018-05-23 05:07:48.641000032+02:00 2018-05-23 05:08:02.177000046+02:00 00:00:13.536000
6712 RAMPDOWN 2018-05-23 05:08:02.177999973+02:00 2018-05-23 05:12:09.263999939+02:00 00:04:07.085999
6713 FILL 2018-05-23 05:12:09.265000105+02:00 2018-05-23 06:27:56.147000074+02:00 01:15:46.881999
6713 SETUP 2018-05-23 05:27:44.332000017+02:00 2018-05-23 05:47:09.704999924+02:00 00:19:25.372999
6713 INJPROT 2018-05-23 05:47:09.706000090+02:00 2018-05-23 05:57:04.144000053+02:00 00:09:54.437999
6713 INJPHYS 2018-05-23 05:57:04.144999981+02:00 2018-05-23 06:27:56.147000074+02:00 00:30:52.002000
6714 FILL 2018-05-23 06:27:56.148000002+02:00 2018-05-23 21:52:48.005000114+02:00 15:24:51.857000
6714 INJPROT 2018-05-23 06:28:16.176000118+02:00 2018-05-23 06:31:35.838000059+02:00 00:03:19.661999
6714 INJPHYS 2018-05-23 06:31:35.838999987+02:00 2018-05-23 07:00:19.354000092+02:00 00:28:43.515000
6714 PRERAMP 2018-05-23 07:00:19.355000019+02:00 2018-05-23 07:03:33.286999941+02:00 00:03:13.931999
6714 RAMP 2018-05-23 07:03:33.288000107+02:00 2018-05-23 07:23:47.737999916+02:00 00:20:14.449999
6714 FLATTOP 2018-05-23 07:23:47.739000082+02:00 2018-05-23 07:27:13.437000036+02:00 00:03:25.697999
6714 SQUEEZE 2018-05-23 07:27:13.437999964+02:00 2018-05-23 07:38:08.301000118+02:00 00:10:54.863000
6714 ADJUST 2018-05-23 07:38:08.302000046+02:00 2018-05-23 07:49:04.497999907+02:00 00:10:56.195999
6714 STABLE 2018-05-23 07:49:04.499000072+02:00 2018-05-23 21:48:11.851999998+02:00 13:59:07.352999
6714 BEAMDUMP 2018-05-23 21:48:11.852999926+02:00 2018-05-23 21:48:40.487999916+02:00 00:00:28.634999
6714 RAMPDOWN 2018-05-23 21:48:40.489000082+02:00 2018-05-23 21:52:48.005000114+02:00 00:04:07.516000
6715 FILL 2018-05-23 21:52:48.006000042+02:00 2018-05-24 00:47:48.658999920+02:00 02:55:00.652999
6715 SETUP 2018-05-23 22:23:27.148999929+02:00 2018-05-23 23:28:07.967999935+02:00 01:04:40.819000
6715 INJPROT 2018-05-23 23:28:07.969000101+02:00 2018-05-24 00:20:01.910000086+02:00 00:51:53.940999
6715 INJPHYS 2018-05-24 00:20:01.911000013+02:00 2018-05-24 00:47:48.658999920+02:00 00:27:46.747999
6716 FILL 2018-05-24 00:47:48.660000086+02:00 2018-05-24 06:15:31.756000042+02:00 05:27:43.095999
6716 INJPROT 2018-05-24 00:48:01.979000092+02:00 2018-05-24 04:11:03.315000057+02:00 03:23:01.335999
6716 NOBEAM 2018-05-24 04:11:03.315999985+02:00 2018-05-24 06:15:29.599999905+02:00 02:04:26.283999
6716 CYCLING 2018-05-24 06:15:29.601000071+02:00 2018-05-24 06:15:31.756000042+02:00 00:00:02.154999
6717 FILL 2018-05-24 06:15:31.756999969+02:00 2018-05-24 20:49:18.076999903+02:00 14:33:46.319999
6717 NOBEAM 2018-05-24 07:37:40.792000055+02:00 2018-05-24 20:49:18.076999903+02:00 13:11:37.284999

925 rows × 4 columns

LHCFillsByNumber function

One can select the LHC fill by fill number.

dfFILL=importData.LHCFillsByNumber(6611)
dfFILL
mode startTime endTime duration
6611 FILL 2018-04-24 16:57:54.688999891+00:00 2018-04-25 07:15:18.815999985+00:00 14:17:24.127000
6611 SETUP 2018-04-24 17:16:51.466000080+00:00 2018-04-24 18:56:27.641000032+00:00 01:39:36.174999
6611 INJPROT 2018-04-24 18:56:27.641999960+00:00 2018-04-24 19:13:20.552999973+00:00 00:16:52.911000
6611 INJPHYS 2018-04-24 19:13:20.553999901+00:00 2018-04-24 19:52:14.631000042+00:00 00:38:54.077000
6611 PRERAMP 2018-04-24 19:52:14.631999969+00:00 2018-04-24 19:54:30.025000095+00:00 00:02:15.393000
6611 RAMP 2018-04-24 19:54:30.026000023+00:00 2018-04-24 20:14:44.450000048+00:00 00:20:14.424000
6611 FLATTOP 2018-04-24 20:14:44.450999975+00:00 2018-04-24 20:17:40.210000038+00:00 00:02:55.759000
6611 SQUEEZE 2018-04-24 20:17:40.210999966+00:00 2018-04-24 20:28:38.168999910+00:00 00:10:57.957999
6611 ADJUST 2018-04-24 20:28:38.170000076+00:00 2018-04-24 20:35:27.440000057+00:00 00:06:49.269999
6611 STABLE 2018-04-24 20:35:27.440999985+00:00 2018-04-25 06:00:49.973999977+00:00 09:25:22.532999
6611 ADJUST 2018-04-25 06:00:49.974999905+00:00 2018-04-25 07:09:16.572999954+00:00 01:08:26.598000
6611 BEAMDUMP 2018-04-25 07:09:16.573999882+00:00 2018-04-25 07:10:50.769000053+00:00 00:01:34.195000
6611 RAMPDOWN 2018-04-25 07:10:50.769999981+00:00 2018-04-25 07:15:18.815999985+00:00 00:04:28.046000
dfFILL=importData.LHCFillsByNumber([6611,6666])
dfFILL
mode startTime endTime duration
6611 FILL 2018-04-24 16:57:54.688999891+00:00 2018-04-25 07:15:18.815999985+00:00 14:17:24.127000
6611 SETUP 2018-04-24 17:16:51.466000080+00:00 2018-04-24 18:56:27.641000032+00:00 01:39:36.174999
6611 INJPROT 2018-04-24 18:56:27.641999960+00:00 2018-04-24 19:13:20.552999973+00:00 00:16:52.911000
6611 INJPHYS 2018-04-24 19:13:20.553999901+00:00 2018-04-24 19:52:14.631000042+00:00 00:38:54.077000
6611 PRERAMP 2018-04-24 19:52:14.631999969+00:00 2018-04-24 19:54:30.025000095+00:00 00:02:15.393000
6611 RAMP 2018-04-24 19:54:30.026000023+00:00 2018-04-24 20:14:44.450000048+00:00 00:20:14.424000
6611 FLATTOP 2018-04-24 20:14:44.450999975+00:00 2018-04-24 20:17:40.210000038+00:00 00:02:55.759000
6611 SQUEEZE 2018-04-24 20:17:40.210999966+00:00 2018-04-24 20:28:38.168999910+00:00 00:10:57.957999
6611 ADJUST 2018-04-24 20:28:38.170000076+00:00 2018-04-24 20:35:27.440000057+00:00 00:06:49.269999
6611 STABLE 2018-04-24 20:35:27.440999985+00:00 2018-04-25 06:00:49.973999977+00:00 09:25:22.532999
6611 ADJUST 2018-04-25 06:00:49.974999905+00:00 2018-04-25 07:09:16.572999954+00:00 01:08:26.598000
6611 BEAMDUMP 2018-04-25 07:09:16.573999882+00:00 2018-04-25 07:10:50.769000053+00:00 00:01:34.195000
6611 RAMPDOWN 2018-04-25 07:10:50.769999981+00:00 2018-04-25 07:15:18.815999985+00:00 00:04:28.046000
6666 FILL 2018-05-10 19:35:38.635999918+00:00 2018-05-11 10:29:36.956000090+00:00 14:53:58.320000
6666 INJPROT 2018-05-10 19:35:51.628000021+00:00 2018-05-10 19:39:39.914999962+00:00 00:03:48.286999
6666 INJPHYS 2018-05-10 19:39:39.915999889+00:00 2018-05-10 20:25:05.993000031+00:00 00:45:26.077000
6666 PRERAMP 2018-05-10 20:25:05.993999958+00:00 2018-05-10 20:33:55.484999895+00:00 00:08:49.490999
6666 RAMP 2018-05-10 20:33:55.486000061+00:00 2018-05-10 20:54:09.895999908+00:00 00:20:14.409999
6666 FLATTOP 2018-05-10 20:54:09.897000074+00:00 2018-05-10 20:56:43.071000099+00:00 00:02:33.174000
6666 SQUEEZE 2018-05-10 20:56:43.072000027+00:00 2018-05-10 21:07:37.651000023+00:00 00:10:54.578999
6666 ADJUST 2018-05-10 21:07:37.651999950+00:00 2018-05-10 21:13:16.588999987+00:00 00:05:38.937000
6666 STABLE 2018-05-10 21:13:16.589999914+00:00 2018-05-11 09:56:28.926000118+00:00 12:43:12.336000
6666 ADJUST 2018-05-11 09:56:28.927000046+00:00 2018-05-11 10:22:51.443000078+00:00 00:26:22.516000
6666 BEAMDUMP 2018-05-11 10:22:51.444000006+00:00 2018-05-11 10:24:59.108000040+00:00 00:02:07.664000
6666 BEAMDUMP 2018-05-11 10:24:59.108999968+00:00 2018-05-11 10:25:15.894000053+00:00 00:00:16.785000
6666 RAMPDOWN 2018-05-11 10:25:15.894999981+00:00 2018-05-11 10:29:36.956000090+00:00 00:04:21.061000

LHCInstant

To retrieve the status of the LHC (fill and beammode at a specific time).

t1 = pd.Timestamp('2018-05-22 02:10:15', tz='CET')
importData.LHCInstant(t1)
mode startTime endTime duration
6711 INJPHYS 2018-05-21 23:23:11.940000057+00:00 2018-05-22 00:27:06.229000092+00:00 01:03:54.289000

LHCCals2pd

To collect datas for specific LHC fills or beam modes.

importData.LHCCals2pd?
importData.LHCCals2pd(['%.RQX.%1:I_MEAS'],[6278],['RAMP','FLATTOP'],fill_column=True, beamMode_column=True)
RPHFC.UL14.RQX.L1:I_MEAS RPHFC.UL16.RQX.R1:I_MEAS fill mode
2017-10-06 21:25:06.059999943+00:00 NaN 407.31 6278 RAMP
2017-10-06 21:25:06.079999924+00:00 407.37 NaN 6278 RAMP
2017-10-06 21:25:06.559999943+00:00 NaN 407.38 6278 RAMP
2017-10-06 21:25:06.579999924+00:00 407.45 NaN 6278 RAMP
2017-10-06 21:25:10.059999943+00:00 NaN 408.06 6278 RAMP
2017-10-06 21:25:10.079999924+00:00 408.12 NaN 6278 RAMP
2017-10-06 21:25:10.559999943+00:00 NaN 408.18 6278 RAMP
2017-10-06 21:25:10.579999924+00:00 408.24 NaN 6278 RAMP
2017-10-06 21:25:13.059999943+00:00 NaN 408.82 6278 RAMP
2017-10-06 21:25:13.079999924+00:00 408.89 NaN 6278 RAMP
2017-10-06 21:25:13.559999943+00:00 NaN 408.96 6278 RAMP
2017-10-06 21:25:13.579999924+00:00 409.02 NaN 6278 RAMP
2017-10-06 21:25:15.559999943+00:00 NaN 409.57 6278 RAMP
2017-10-06 21:25:15.579999924+00:00 409.64 NaN 6278 RAMP
2017-10-06 21:25:16.059999943+00:00 NaN 409.74 6278 RAMP
2017-10-06 21:25:16.079999924+00:00 409.81 NaN 6278 RAMP
2017-10-06 21:25:17.559999943+00:00 NaN 410.26 6278 RAMP
2017-10-06 21:25:17.579999924+00:00 410.33 NaN 6278 RAMP
2017-10-06 21:25:18.059999943+00:00 NaN 410.44 6278 RAMP
2017-10-06 21:25:18.079999924+00:00 410.51 NaN 6278 RAMP
2017-10-06 21:25:19.559999943+00:00 NaN 411.02 6278 RAMP
2017-10-06 21:25:19.579999924+00:00 411.09 NaN 6278 RAMP
2017-10-06 21:25:20.059999943+00:00 NaN 411.22 6278 RAMP
2017-10-06 21:25:20.079999924+00:00 411.29 NaN 6278 RAMP
2017-10-06 21:25:21.559999943+00:00 NaN 411.84 6278 RAMP
2017-10-06 21:25:21.579999924+00:00 411.91 NaN 6278 RAMP
2017-10-06 21:25:22.059999943+00:00 NaN 412.07 6278 RAMP
2017-10-06 21:25:22.079999924+00:00 412.14 NaN 6278 RAMP
2017-10-06 21:25:23.559999943+00:00 NaN 412.75 6278 RAMP
2017-10-06 21:25:23.579999924+00:00 412.82 NaN 6278 RAMP
... ... ... ... ...
2017-10-06 21:44:58.559999943+00:00 NaN 6254.74 6278 RAMP
2017-10-06 21:44:58.579999924+00:00 6256.49 NaN 6278 RAMP
2017-10-06 21:44:59.059999943+00:00 NaN 6255.41 6278 RAMP
2017-10-06 21:44:59.079999924+00:00 6257.15 NaN 6278 RAMP
2017-10-06 21:44:59.559999943+00:00 NaN 6256.01 6278 RAMP
2017-10-06 21:44:59.579999924+00:00 6257.75 NaN 6278 RAMP
2017-10-06 21:45:00.059999943+00:00 NaN 6256.54 6278 RAMP
2017-10-06 21:45:00.079999924+00:00 6258.28 NaN 6278 RAMP
2017-10-06 21:45:00.559999943+00:00 NaN 6257.00 6278 RAMP
2017-10-06 21:45:00.579999924+00:00 6258.74 NaN 6278 RAMP
2017-10-06 21:45:01.059999943+00:00 NaN 6257.40 6278 RAMP
2017-10-06 21:45:01.079999924+00:00 6259.14 NaN 6278 RAMP
2017-10-06 21:45:01.559999943+00:00 NaN 6257.73 6278 RAMP
2017-10-06 21:45:01.579999924+00:00 6259.46 NaN 6278 RAMP
2017-10-06 21:46:01.059999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:46:01.079999924+00:00 6260.10 NaN 6278 FLATTOP
2017-10-06 21:46:01.559999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:46:01.579999924+00:00 6260.10 NaN 6278 FLATTOP
2017-10-06 21:47:01.059999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:47:01.079999924+00:00 6260.10 NaN 6278 FLATTOP
2017-10-06 21:47:01.559999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:47:01.579999924+00:00 6260.10 NaN 6278 FLATTOP
2017-10-06 21:48:01.059999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:48:01.079999924+00:00 6260.10 NaN 6278 FLATTOP
2017-10-06 21:48:01.559999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:48:01.579999924+00:00 6260.10 NaN 6278 FLATTOP
2017-10-06 21:49:01.059999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:49:01.079999924+00:00 6260.10 NaN 6278 FLATTOP
2017-10-06 21:49:01.559999943+00:00 NaN 6258.37 6278 FLATTOP
2017-10-06 21:49:01.579999924+00:00 6260.10 NaN 6278 FLATTOP

4742 rows × 4 columns

massiFile2pd function

One can select load massi files in pandas Dataframes.

ATLAS=importData.massiFile2pd('/eos/user/s/sterbini/MD_ANALYSIS/2017/LHC/MD2201/ATLAS_6195.tgz')
ATLAS.tail(3)
FILL Stable Beam Flag Experiment Bunch Luminosity [Hz/ub] P2P luminosity error [Hz/ub] Specific luminosity [Hz/ub] P2P specific luminosity [Hz/ub]
2017-09-13 18:00:43+00:00 6195 0 ATLAS 1490 0.000047 0.0 0.0 0.0
2017-09-13 18:01:43+00:00 6195 0 ATLAS 1490 0.000047 0.0 0.0 0.0
2017-09-13 18:21:26+00:00 6195 0 ATLAS 1490 0.000017 0.0 0.0 0.0

calsCSV2pd function

This allows to convert in pandas dataframes the csv data obtained from CALS.

myFile='/eos/project/l/lhc-lumimod/LuminosityFollowUp/2017/rawdata/fill_bunchbybunch_data_csvs/bunchbybunch_lumi_data_fill_5848.csv'
aux=importData.calsCSV2pd(myFile)
aux.head()
ATLAS:BUNCH_LUMI_INST CMS:BUNCH_LUMI_INST
2017-06-19 15:42:47.745000+00:00 [0.045945592, 0.045026667, 0.046864517, 0.0450... NaN
2017-06-19 15:44:48.104000+00:00 [0.039506007, 0.04899983, 0.041649766, 0.04011... NaN
2017-06-19 15:45:48.287000+00:00 [0.044709116, 0.04869012, 0.0523649, 0.0431779... NaN
2017-06-19 15:47:48.234000+00:00 [0.03982115, 0.039514832, 0.042884354, 0.04870... NaN
2017-06-19 15:48:48.298000+00:00 [0.05175795, 0.04410135, 0.049307834, 0.049614... NaN
# to retrieve directly from CALS. This is for checks and in general is slower
aux=importData.cals2pd(aux.columns,t1=aux.index[0],t2=aux.index[2])
aux
ATLAS:BUNCH_LUMI_INST CMS:BUNCH_LUMI_INST
2017-06-19 15:42:47.745000124+00:00 [0.045945592, 0.045026667, 0.046864517, 0.0450... NaN
2017-06-19 15:44:48.104000092+00:00 [0.039506007, 0.04899983, 0.041649766, 0.04011... NaN
2017-06-19 15:45:48.286999941+00:00 [0.044709116, 0.04869012, 0.0523649, 0.0431779... NaN
myFile='/eos/project/l/lhc-lumimod/LuminosityFollowUp/2017/rawdata/fill_basic_data_csvs/basic_data_fill_5706.csv'
aux=importData.calsCSV2pd(myFile)
aux.head()
LHC.BCTDC.A6R4.B1:BEAM_INTENSITY LHC.BCTDC.A6R4.B2:BEAM_INTENSITY LHC.BSRA.US45.B1:ABORT_GAP_ENERGY LHC.BSRA.US45.B2:ABORT_GAP_ENERGY
2017-05-25 13:01:13+00:00 2.410000e+09 590000000.0 NaN NaN
2017-05-25 13:01:13.736000+00:00 NaN NaN 450.0 450.0
2017-05-25 13:01:14+00:00 8.600000e+08 390000000.0 NaN NaN
2017-05-25 13:01:14.736000+00:00 NaN NaN 450.0 450.0
2017-05-25 13:01:15+00:00 1.300000e+09 -150000000.0 NaN NaN

mat2dict function

Import a mat file in a mat_struct (very similar to a dict) that can be easily browsed.

aux=importData.mat2dict('/eos/user/s/sterbini/MD_ANALYSIS/2016/MD1780_80b/2016.10.26.22.23.42.135.mat')
aux.CPS_BLM
<scipy.io.matlab.mio5_params.mat_struct at 0x7f84505ba128>

mat2pd function

aux=importData.mat2pd(['CPS_BLM.Acquisition.value.lastLosses'],['/eos/user/s/sterbini/MD_ANALYSIS/2016/MD1780_80b/2016.10.26.22.23.42.135.mat',
                                                   '/eos/user/s/sterbini/MD_ANALYSIS/2016/MD1780_80b/2016.10.26.22.23.06.147.mat'])
aux
CPS_BLM.Acquisition.value.lastLosses matlabFilePath
2016-10-26 20:22:27.100000+00:00 [53, 30, 95, 29, 12, 18, 3, 3, 12, 4, 2, 6, 0,... /eos/user/s/sterbini/MD_ANALYSIS/2016/MD1780_8...
2016-10-26 20:23:03.100000+00:00 [58, 32, 93, 29, 12, 18, 3, 3, 13, 4, 2, 6, 0,... /eos/user/s/sterbini/MD_ANALYSIS/2016/MD1780_8...

tfs2pd function

With this funtion we can import the the MADX TFS file in a pandas Dataframe.

aux=importData.tfs2pd('/eos/user/s/sterbini/MD_ANALYSIS/2018/LHC MD Optics/collisionAt25cm_180urad/lhcb1_thick.survey')
display(aux.transpose())
aux.iloc[0]['TABLE'].head()
The column SLOT_ID is empty.
The column ASSEMBLY_ID is empty.
/eos/user/s/sterbini/MD_ANALYSIS/2018/LHC MD Optics/collisionAt25cm_180urad/lhcb1_thick.survey
DATE 22/03/18
ORIGIN 5.03.07
TIME 22.30.11
TITLE no-title
TYPE SURVEY
TABLE NAME KEYWORD ...
NAME KEYWORD S L ANGLE X Y Z THETA PHI PSI GLOBALTILT MECH_SEP V_POS
0.000 IP1 MARKER 0.000 0.000 0.0 0.0 0.0 0.000 0.0 0.0 0.0 0.0 0.0 0.0
3.000 MBAS2.1R1 SOLENOID 3.000 3.000 0.0 0.0 0.0 3.000 0.0 0.0 0.0 0.0 0.0 0.0
19.050 DRIFT_0 DRIFT 19.050 16.050 0.0 0.0 0.0 19.050 0.0 0.0 0.0 0.0 0.0 0.0
20.850 TAS.1R1 RCOLLIMATOR 20.850 1.800 0.0 0.0 0.0 20.850 0.0 0.0 0.0 0.0 0.0 0.0
21.564 DRIFT_1 DRIFT 21.564 0.714 0.0 0.0 0.0 21.564 0.0 0.0 0.0 0.0 0.0 0.0
aux=importData.tfs2pd('/eos/user/s/sterbini/MD_ANALYSIS/2018/LHC MD Optics/collisionAt25cm_180urad/lhcb1_thick.twiss')
aux.iloc[0]['TABLE'].head()
NAME PARENT KEYWORD S L X Y PX PY BETX BETY ALFX ALFY MUX MUY DX DY DPX DPY
0.000 IP1 OMK MARKER 0.000 0.000 -4.874885e-11 -1.062519e-12 -4.666460e-11 0.00018 0.250001 0.250000 -0.000002 1.233730e-07 0.000000 0.000000 0.026209 0.005558 0.01455 0.039107
3.000 MBAS2.1R1 MBAS2 SOLENOID 3.000 3.000 -1.887427e-10 5.400004e-04 -4.666460e-11 0.00018 36.249874 36.249965 -11.999956 -1.199999e+01 0.236767 0.236768 0.069860 0.122338 0.01455 0.039107
19.050 DRIFT_0 DRIFT DRIFT 19.050 16.050 -9.377096e-10 3.429003e-03 -4.666460e-11 0.00018 1451.854515 1451.858614 -76.199710 -7.619993e+01 0.247911 0.247911 0.303393 0.747112 0.01455 0.039107
20.850 TAS.1R1 TAS RCOLLIMATOR 20.850 1.800 -1.021706e-09 3.753003e-03 -4.666460e-11 0.00018 1739.133423 1739.138340 -83.399683 -8.339992e+01 0.248091 0.248092 0.329583 0.817180 0.01455 0.039107
21.564 DRIFT_1 DRIFT DRIFT 21.564 0.714 -1.055024e-09 3.881523e-03 -4.666460e-11 0.00018 1860.267346 1860.272608 -86.255672 -8.625592e+01 0.248155 0.248155 0.339972 0.844974 0.01455 0.039107
aux=importData.tfs2pd(['/eos/user/s/sterbini/MD_ANALYSIS/2018/LHC MD Optics/collisionAt25cm_180urad/lhcb1_thick.survey','/eos/user/s/sterbini/MD_ANALYSIS/2018/LHC MD Optics/collisionAt25cm_180urad/lhcb1_thick.twiss'])
The column SLOT_ID is empty.
The column ASSEMBLY_ID is empty.
aux.iloc[0]['TABLE'].index
Float64Index([       0.0,        3.0,      19.05,      20.85,     21.564,
                  21.564,     21.564,      21.62,      21.62,     21.724,
              ...
              26637.2482, 26637.2482, 26637.3042, 26637.3042, 26637.3042,
              26638.0042, 26639.8042, 26655.8832, 26658.8832, 26658.8832],
             dtype='float64', length=13283)

From dotdict to a simple pandas Dataframe.

We can transform a dotdict in a pandas Dataframe.

from cl2pd import dotdict
a=dotdict.dotdict()
np=importData.np
for i in {'B1'}:
    a[i]=dotdict.dotdict()
    for j in range(10):
        a[i]['b'+str(j)]=np.random.randn(10)

for i in {'B2'}:
    a[i]=dotdict.dotdict()
    for j in range(20):
        a[i]['b'+str(j)]=np.random.randn(10)
pd.DataFrame(a)
B1 B2
b0 [0.6467321347905923, -1.0370676505843401, -1.5... [1.5270813764876534, 1.2100639941492304, -1.11...
b1 [0.32969179854420916, 0.7858822252604646, -0.3... [1.3064310857233223, -0.3359487143927113, 0.66...
b10 NaN [-0.6957631261643009, 0.32370222762197626, 2.8...
b11 NaN [-0.779053908334011, 1.506635143971471, -0.233...
b12 NaN [2.0691097429781133, -1.528249449203943, 0.477...
b13 NaN [-0.3705018424449959, -1.1877717848803848, -1....
b14 NaN [-0.9949066537823384, -0.35825900064144167, -0...
b15 NaN [0.21650090161488317, -0.5518600501885274, 0.5...
b16 NaN [-0.6802440968350015, 0.7573210133031285, 0.53...
b17 NaN [-2.2167781994862197, 0.6851372659010567, -0.0...
b18 NaN [-1.7360108910276053, 0.3803327796866624, -1.6...
b19 NaN [0.9476187485046135, 0.640806434726803, 0.5297...
b2 [-1.7182497708209206, 0.2865187814502491, -0.7... [0.7847226477419496, -0.3710598283057992, -0.0...
b3 [-1.469273880140067, 0.4212359180496405, 1.037... [0.05188168046413233, 1.4355051492894526, 2.11...
b4 [0.007974681615498608, -0.2610696151209144, 0.... [0.9520874475610571, -1.2910502451899022, -1.7...
b5 [0.49349416323479206, 0.8088796123324145, 0.19... [-0.44431802009622257, 0.4787313869654191, 0.1...
b6 [0.5405258900294131, -0.5084350523854257, 0.14... [-0.7898382541838375, 0.11480202852339073, -0....
b7 [-0.11305147458494907, 0.03314302262877794, -0... [0.896925953242836, -0.14222940386632202, 0.18...
b8 [0.9405878186855667, 1.1799075215016352, 1.087... [1.3164852160372085, -1.1817663457608947, 0.03...
b9 [-0.11927058554763645, -0.0652152524769821, 0.... [-0.4518129428607217, -0.16586580625802247, -0...

Concat check

variables=['LHC.BCTDC.A6R4.B1:BEAM_INTENSITY']
startTime = pd.Timestamp('2017-10-01 17:30', tz='CET')
endTime = pd.Timestamp('2017-10-01 17:31', tz='CET')
raw_data1 = importData.cals2pd(variables,startTime,endTime)
variables=['LHC.BCTDC.A6R4.B2:BEAM_INTENSITY']
raw_data2 = importData.cals2pd(variables,startTime,endTime)
pd.merge(raw_data1,raw_data2,left_index=True,right_index=True).head()
LHC.BCTDC.A6R4.B1:BEAM_INTENSITY LHC.BCTDC.A6R4.B2:BEAM_INTENSITY
2017-10-01 15:30:00+00:00 1.470000e+09 270000000.0
2017-10-01 15:30:01+00:00 1.810000e+09 310000000.0
2017-10-01 15:30:02+00:00 1.690000e+09 -310000000.0
2017-10-01 15:30:03+00:00 1.990000e+09 -720000000.0
2017-10-01 15:30:04+00:00 1.270000e+09 220000000.0
# Interesting checks 
# pd.Timestamp('2018-03-25 02:00', tz='CET')      #this date does not exist, so the function returns an error.
# pd.Timestamp('2015-06-30 23:59:60', tz='UTC')   #this date does exist (leap second) but it is not recognized.

In case of problem with pytimber

if 0:
    import cmmnbuild_dep_manager
    mgr = cmmnbuild_dep_manager.Manager()
    mgr.install('pytimber')
    mgr.resolve()
from cl2pd import variablesDF
variablesDF.LHC
Variable Tag Type On change Description
0 LHC.BOFSU:OFC_ENERGY Energy NUMERIC True Beam energy
1 LHC.BCTFR.A6R4.B1:BEAM_INTENSITY Intensity NUMERIC False B1 intensity
2 LHC.BCTFR.A6R4.B2:BEAM_INTENSITY Intensity NUMERIC False B2 intensity
3 LHC.RUNCONFIG:IP1-XING-V-MURAD Crossing angle NUMERIC True Crossing angle IP1 V
4 LHC.RUNCONFIG:IP2-XING-V-MURAD Crossing angle NUMERIC True Crossing angle IP2 V
5 LHC.RUNCONFIG:IP5-XING-H-MURAD Crossing angle NUMERIC True Crossing angle IP5 H
6 LHC.RUNCONFIG:IP8-XING-H-MURAD Crossing angle NUMERIC True Crossing angle IP8 H
7 LHC.BQM.B1:BUNCH_LENGTH_MEAN Bunch length NUMERIC False Bunch Length B1
8 LHC.BQM.B2:BUNCH_LENGTH_MEAN Bunch length NUMERIC False Bunch Length B2
9 LHC.BQBBQ.CONTINUOUS.B1:TUNE_V Tune NUMERIC False Tune V B1 BBQ
10 LHC.BQBBQ.CONTINUOUS.B2:TUNE_V Tune NUMERIC False Tune V B2 BBQ
11 LHC.BQBBQ.CONTINUOUS_HS.B1:TUNE_V Tune NUMERIC False Most sensitive tune B1 V
12 LHC.BQBBQ.CONTINUOUS_HS.B2:TUNE_V Tune NUMERIC False Most sensitive tune B2 V
13 LHC.BQBBQ.CONTINUOUS.B1:TUNE_H Tune NUMERIC False Tune H B1 BBQ
14 LHC.BQBBQ.CONTINUOUS.B2:TUNE_H Tune NUMERIC False Tune H B2 BBQ
15 LHC.BQBBQ.CONTINUOUS_HS.B1:TUNE_H Tune NUMERIC False Most sensitive tune B1 H
16 LHC.BQBBQ.CONTINUOUS_HS.B2:TUNE_H Tune NUMERIC False Most sensitive tune B2 H
17 LHC.BQBBQ.CONTINUOUS.B1:FFT_DATA_H Tune FFT NUMERIC False Tune FFT H B1
18 LHC.BQBBQ.CONTINUOUS.B1:FFT_DATA_V Tune FFT NUMERIC False Tune FFT V B1
19 LHC.BQBBQ.CONTINUOUS.B2:FFT_DATA_H Tune FFT NUMERIC False Tune FFT H B2
20 LHC.BQBBQ.CONTINUOUS.B2:FFT_DATA_V Tune FFT NUMERIC False Tune FFT V B2
21 LHC.BCTFR.A6R4.B1:BUNCH_INTENSITY Intensity VECTOR NUMERIC False Main FBCT B1
22 LHC.BCTFR.A6R4.B2:BUNCH_INTENSITY Intensity VECTOR NUMERIC False Main FBCT B2
23 LHC.BCTFR.B6R4.B1:BUNCH_INTENSITY Intensity VECTOR NUMERIC False Spare FBCT B1
24 LHC.BCTFR.A6R4.B2:BUNCH_INTENSITY Intensity VECTOR NUMERIC False Spare FBCT B2
25 LHC.BCTFR.A6R4.B1:BUNCH_FILL_PATTERN Filling Pattern NUMERIC False Main FBCT B1
26 LHC.BCTFR.A6R4.B2:BUNCH_FILL_PATTERN Filling Pattern NUMERIC False Main FBCT B2
27 LHC.BCTFR.B6R4.B1:BUNCH_FILL_PATTERN Filling Pattern NUMERIC False Spare FBCT B1
28 LHC.BCTFR.A6R4.B2:BUNCH_FILL_PATTERN Filling Pattern NUMERIC False Spare FBCT B2
29 CMS:BUNCH_LUMI_INST Luminosity VECTOR NUMERIC False CMS bbb luminosity
... ... ... ... ... ...
124 ADTH.SR4.M1.B2:MDSPU_PHASE1 ADT Phase NUMERIC False B2 Hozirontal ADT Phase 1 Mon1 (deg)
125 ADTH.SR4.M1.B2:MDSPU_PHASE2 ADT Phase NUMERIC False B2 Hozirontal ADT Phase 2 Mon1 (deg)
126 ADTH.SR4.M1.B2:MDSPU_PHASE3 ADT Phase NUMERIC False B2 Hozirontal ADT Phase 3 Mon1 (deg)
127 ADTH.SR4.M1.B2:MDSPU_PHASE4 ADT Phase NUMERIC False B2 Hozirontal ADT Phase 4 Mon1 (deg)
128 ADTV.SR4.M1.B1:MDSPU_PHASE1 ADT Phase NUMERIC False B2 Vertical ADT Phase 1 Mon1 (deg)
129 ADTV.SR4.M1.B1:MDSPU_PHASE2 ADT Phase NUMERIC False B2 Vertical ADT Phase 2 Mon1 (deg)
130 ADTV.SR4.M1.B1:MDSPU_PHASE3 ADT Phase NUMERIC False B2 Vertical ADT Phase 3 Mon1 (deg)
131 ADTV.SR4.M1.B1:MDSPU_PHASE4 ADT Phase NUMERIC False B2 Vertical ADT Phase 4 Mon1 (deg)
132 ADTH.SR4.B1:TRANSVERSEACTIVITYMAX_HB1 ADT Activity NUMERIC False B1 Horizontal ADT Transverse Activity (um)
133 ADTV.SR4.B1:TRANSVERSEACTIVITYMAX_VB1 ADT Activity NUMERIC False B1 Vertical ADT Transverse Activity (um)
134 ADTH.SR4.B2:TRANSVERSEACTIVITYMAX_HB2 ADT Activity NUMERIC False B2 Horizontal ADT Transverse Activity (um)
135 ADTV.SR4.B2:TRANSVERSEACTIVITYMAX_VB2 ADT Activity NUMERIC False B2 Vertical ADT Transverse Activity (um)
136 LHC.BOFSU:DEFLECTIONS_H Orbit VECTORNUMERIC False COD deflections in horizontal plane (urad)
137 LHC.BOFSU:DEFLECTIONS_V Orbit VECTORNUMERIC False COD deflections in vertical plane (urad)
138 LHC.BPTUH.A4L5.B1:CALIBLINEARPOS Orbit NUMERIC False Upstream Horizontal TCT.A4L5.B1 Calibrated Lin...
139 LHC.BPTUV.A4L5.B1:CALIBLINEARPOS Orbit NUMERIC False Upstream Vertical TCT.A4L5.B1 Calibrated Linea...
140 LHC.BPTUH.A4R5.B2:CALIBLINEARPOS Orbit NUMERIC False Upstream Horizontal TCT.A4R5.B2 Calibrated Lin...
141 LHC.BPTUV.A4R5.B2:CALIBLINEARPOS Orbit NUMERIC False Upstream Vertical TCT.A4R5.B2 Calibrated Linea...
142 LHC.BPTDH.A4L5.B1:CALIBLINEARPOS Orbit NUMERIC False Downstream Horizontal TCT.A4L5.B1 Calibrated L...
143 LHC.BPTDV.A4L5.B1:CALIBLINEARPOS Orbit NUMERIC False Downstream Vertical TCT.A4L5.B1 Calibrated Lin...
144 LHC.BPTDH.A4R5.B2:CALIBLINEARPOS Orbit NUMERIC False Downstream Horizontal TCT.A4R5.B2 Calibrated L...
145 LHC.BPTDV.A4R5.B2:CALIBLINEARPOS Orbit NUMERIC False Downstream Vertical TCT.A4R5.B2 Calibrated Lin...
146 LHC.BPTUH.A4L1.B1:CALIBLINEARPOS Orbit NUMERIC False Upstream Horizontal TCT.A4L1.B1 Calibrated Lin...
147 LHC.BPTUV.A4L1.B1:CALIBLINEARPOS Orbit NUMERIC False Upstream Vertical TCT.A4L1.B1 Calibrated Linea...
148 LHC.BPTUH.A4R1.B2:CALIBLINEARPOS Orbit NUMERIC False Upstream Horizontal TCT.A4R1.B2 Calibrated Lin...
149 LHC.BPTUV.A4R1.B2:CALIBLINEARPOS Orbit NUMERIC False Upstream Vertical TCT.A4R1.B2 Calibrated Linea...
150 LHC.BPTDH.A4L1.B1:CALIBLINEARPOS Orbit NUMERIC False Downstream Horizontal TCT.A4L1.B1 Calibrated L...
151 LHC.BPTDV.A4L1.B1:CALIBLINEARPOS Orbit NUMERIC False Downstream Vertical TCT.A4L1.B1 Calibrated Lin...
152 LHC.BPTDH.A4R1.B2:CALIBLINEARPOS Orbit NUMERIC False Downstream Horizontal TCT.A4R1.B2 Calibrated L...
153 LHC.BPTDV.A4R1.B2:CALIBLINEARPOS Orbit NUMERIC False Downstream Vertical TCT.A4R1.B2 Calibrated Lin...

154 rows × 5 columns

LPC website

A rich source of information is the LPC website. We will show how to import its data in pd dataframes.

Getting the filling table

def LPCFillTable(year=2018):
    '''
    It return the LPC Fill Table of the specified year.
    Note that the fill table is available only for a limited number of year.
    '''
    import urllib.request, json 
    with urllib.request.urlopen('http://lpc.web.cern.ch/cgi-bin/fillTableReader.py?action=load&year='+str(year)) as url:
        data = json.loads(url.read().decode())
    if len(data['data'])>0: 
        myDF=pd.DataFrame(data['data']).transpose()[['start_sb','length_sb','type','scheme','end','ta','fl','remarks']]
        return myDF.sort_index()
    else:
        return pd.DataFrame()
LPCFillTable()
start_sb length_sb type scheme end ta fl remarks
6469 2018-03-22 18:47:32 0:50:55 special alternating R1 R2 pilot operator True True handshake tests
6570 2018-04-17 12:54:14 6:36:36 rampup Single_3b_2_2_2 operator True True First STABLE BEAMS in 2018. At the start befor...
6573 2018-04-18 00:29:29 6:20:49 rampup Single_3b_2_2_2 operator True True Second fill with crossing angle leveling and s...
6574 2018-04-18 22:28:06 11:33:43 rampup Single_12b_8_8_8_2018 operator True True Last fill of first intensity step. At the end ...
6579 2018-04-20 00:22:38 11:58:16 rampup 25ns_75b_62_32_62_12bpi_9inj RF True True First fill with trains. At the end ß* leveling...
6583 2018-04-20 19:48:23 5:58:46 rampup 25ns_75b_62_32_62_12bpi_9inj others True True Dump by COD interlock. Lumi was lower since th...
6584 2018-04-21 04:04:20 6:50:59 rampup no_value operator True True ß* leveling in SB. Lumi in LHCb moved the same...
6592 2018-04-21 23:15:46 2:54:31 rampup Single_10b_9_2_2_BSRT_Calib_RampUp operator True True Emittance between 1.5 and 4.5 µm
6594 2018-04-22 09:29:22 9:14:28 rampup 25ns_339b_326_206_266_12bpi_31inj operator True True Was stopped due to access request of ATLAS for...
6595 2018-04-22 22:14:52 9:04:39 rampup 25ns_339b_326_206_266_12bpi_31inj operator True True CMS took some low µ data. After this fill one ...
6611 2018-04-24 22:35:27 9:25:22 rampup 25ns_315b_302_237_240_48bpi_11inj operator True True Loss spike due to 16L2 reached 40% during ramp...
6612 2018-04-25 14:16:59 8:36:17 rampup 25ns_603b_590_524_542_48bpi_17inj operator True True
6613 2018-04-26 01:23:52 0:29:22 rampup 25ns_603b_590_524_542_48bpi_17inj others True True BTV interlock. Crate exchanged
6614 2018-04-26 03:46:32 5:24:46 rampup 25ns_603b_590_524_542_48bpi_17inj pwr converter True True RCBCVS5.R2B1
6615 2018-04-26 19:54:06 5:11:26 rampup 25ns_603b_590_526_547_96bpi_13inj operator True True operator
6616 2018-04-27 04:44:15 8:10:27 rampup 25ns_987b_974_876_912_96bpi_17inj operator True True ALICE lower solenoid field (12kA instead of 30kA)
6617 2018-04-27 16:40:40 8:40:53 rampup 25ns_987b_974_878_917_144bpi_13inj operator False True During first filling trial one bunch was missi...
6618 2018-04-28 05:05:13 4:03:42 rampup 25ns_987b_974_878_917_144bpi_13inj operator True True
6620 2018-04-28 21:31:28 5:12:16 rampup 25ns_1227b_1214_1054_1102_144bpi_14inj 16L2 True True ...number one... Beam 2 dumped. ADT B1 H was off
6621 2018-04-29 05:17:00 10:19:29 rampup 25ns_1227b_1214_1054_1102_144bpi_14inj operator False True IP2 IP8 lumi was changing when crossing angle ...
6624 2018-04-30 01:52:55 11:12:05 rampup 25ns_1227b_1214_1054_1102_144bpi_14inj operator False True
6628 2018-04-30 21:07:28 12:01:33 physics 25ns_1551b_1538_1404_1467_144bpi_16inj operator False True L=1.3e34; first fill with continuous crossing ...
6629 2018-05-01 12:07:32 10:38:32 physics 25ns_1887b_1874_1694_1772_144bpi_19inj pwr converter False True L=1.6e34; trip of orbit corrector; afterwards ...
6633 2018-05-02 06:08:50 0:47:31 physics 25ns_1887b_1874_1694_1772_144bpi_19inj RF False True During filling the first time uncaptured beam ...
6636 2018-05-02 11:41:44 2:41:30 physics 25ns_1887b_1874_1694_1772_144bpi_19inj pwr converter False False Orbit Corrector trip; communication fault.&nbs...
6638 2018-05-02 23:30:33 14:20:52 physics 25ns_2175b_2162_1874_1964_144bpi_19inj operator False False Abort gap keeper changed to 32811. RF voltage ...
6639 2018-05-03 16:57:34 11:54:41 physics 25ns_2319b_2306_1964_2060_144bpi_22inj pwr converter False False Lower currents since settings in Injectors wer...
6640 2018-05-04 10:07:21 9:09:26 physics 25ns_2460b_2448_2052_2154_144bpi_19injv2 operator False False Lower intenity due to PS RF issue.
6641 2018-05-04 21:34:11 7:06:39 physics 25ns_2460b_2448_2052_2154_144bpi_19injv2 QPS False False trip of RCBXH3.L5 and RCBXH3.L5 triggered by Q...
6642 2018-05-05 07:07:09 3:56:27 physics 25ns_2460b_2448_2052_2154_144bpi_19injv2 others False False BLM dumped the beam (SUE candidarw)
... ... ... ... ... ... ... ... ...
7449 2018-11-16 18:31:13 7:23:15 ions 100_150ns_648Pb_620_619_52_36bpi_20inj_V2 operator False False
7450 2018-11-17 04:30:21 7:20:12 ions 100_150ns_648Pb_620_619_52_36bpi_20inj_V2 operator False False After wards fill 7451 was again lost due to lo...
7453 2018-11-18 07:36:17 11:14:01 ions 100_150ns_648Pb_620_619_52_36bpi_20inj_V2 operator False False kept long due to Linac-III source and LEIR pro...
7454 2018-11-19 00:35:24 11:21:55 ions 100_150ns_648Pb_620_619_52_36bpi_20inj_V2 operator False False
7456 2018-11-19 22:21:10 6:51:04 ions 100_150ns_648Pb_620_619_52_36bpi_20inj_V2 operator False False
7457 2018-11-20 11:03:57 7:16:12 ions 100_150ns_648Pb_620_619_52_36bpi_20inj_V2 operator False False
7459 2018-11-21 00:09:47 0:09:48 ions 75_150ns_460Pb_460_456_304_36bpi_17inj operator False False First ramp up fill with reversed polarity in A...
7460 2018-11-21 02:41:29 4:51:48 ions 75_150ns_460Pb_460_456_304_36bpi_17inj operator False False Second ramp up fill with reversed polarity in ...
7464 2018-11-22 03:58:13 6:31:19 ions 75_150ns_460Pb_460_456_304_42bpi_13inj operator False False Third ramp up fill.
7466 2018-11-22 19:08:59 8:37:46 ions 75_150ns_670Pb_607_630_384_42bpi_18inj operator False False Levelling IP 1&amp;5 at 3.5e27. Reduced fillin...
7467 2018-11-23 09:58:59 7:58:52 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False Levelling IP 1&amp;5 at 4e27.
7468 2018-11-23 20:43:11 9:50:12 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False Levelling IP 1&amp;5 at 5e27.
7471 2018-11-24 14:35:23 8:07:52 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7472 2018-11-25 01:25:01 7:50:12 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7473 2018-11-25 11:49:05 9:48:03 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False Problem with inner detector power supply in AL...
7474 2018-11-26 02:13:25 5:55:26 ions 75_150ns_733Pb_733_702_468_42bpi_20inj pwr converter False False Power converters losing communication
7475 2018-11-26 18:06:32 0:38:36 ions 75_150ns_733Pb_733_702_468_42bpi_20inj QPS False False Spurious QPS trip in RB of S12
7477 2018-11-26 23:27:29 8:50:01 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7480 2018-11-28 00:10:18 7:53:18 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7481 2018-11-28 11:00:42 8:23:53 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7482 2018-11-28 23:10:05 0:31:41 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False Losses in IR7 with 10Hz oscillations
7483 2018-11-29 02:33:11 13:01:29 ions 100_150ns_648Pb_620_619_52_36bpi_20inj_V2 operator False False VDM repetition for ALICE in shadow of&nbsp;
7485 2018-11-29 21:29:08 7:51:20 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False Slow filling
7486 2018-11-30 09:38:54 7:35:52 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7487 2018-11-30 20:04:57 8:29:44 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7488 2018-12-01 07:40:42 8:05:50 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7489 2018-12-01 18:27:18 0:44:40 ions 75_150ns_733Pb_733_702_468_42bpi_20inj others False False Vacuum
7490 2018-12-01 21:37:48 8:31:35 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False
7491 2018-12-02 08:48:41 4:50:14 ions 75_150ns_733Pb_733_702_468_42bpi_20inj RF False False RF
7492 2018-12-02 15:52:49 1:18:06 ions 75_150ns_733Pb_733_702_468_42bpi_20inj operator False False

291 rows × 8 columns

Luminosity data

def LPCLumiData(source='http://lpc-afs.web.cern.ch/lpc-afs/LHC/2018/luminosity_data_ions.txt'):
    import requests

    response = requests.get('http://lpc-afs.web.cern.ch/lpc-afs/LHC/2018/luminosity_data_ions.txt')
    data = response.text
    aux=(pd.DataFrame([x.split() for x in data.split('\n')[2:]], columns=['fill','tstart_UNIX_UTC','tstop_UNIX_UTC','ATLAS_lui_inverse_ub',\
                                                                    'CMS_lui_inverse_ub', 'LHCb_lui_inverse_ub', 'ALICE_lui_inverse_ub','ATLAS_plu_Hz_inverse_ub',\
                                                                    'CMS_plu_Hz_inverse_ub', 'LHCb_plu_Hz_inverse_ub', 'ALICE_plu_Hz_inverse_ub']))
    aux=aux.set_index('fill')
    aux.index.name=None
    return aux
LPCLumiData(source='http://lpc-afs.web.cern.ch/lpc-afs/LHC/2018/luminosity_data_ions.txt')
tstart_UNIX_UTC tstop_UNIX_UTC ATLAS_lui_inverse_ub CMS_lui_inverse_ub LHCb_lui_inverse_ub ALICE_lui_inverse_ub ATLAS_plu_Hz_inverse_ub CMS_plu_Hz_inverse_ub LHCb_plu_Hz_inverse_ub ALICE_plu_Hz_inverse_ub
7427 1541708363 1541733698 1 1 0 2 1.510e-04 1.720e-04 3.200e-05 2.065e-04
7428 1541741102 1541752703 2 2 0 2 1.320e-04 1.290e-04 2.900e-05 8.785e-05
7433 1541805329 1541820593 10 10 0 6 8.220e-04 8.150e-04 2.800e-05 4.479e-04
7435 1541864777 1541885107 21 21 1 12 9.940e-04 9.870e-04 3.000e-05 5.904e-04
7436 1541897140 1541917561 46 47 2 26 2.461e-03 2.445e-03 8.700e-05 9.689e-04
7437 1541933644 1541947812 64 64 2 34 2.350e-03 2.360e-03 7.600e-05 9.662e-04
7438 1541962710 1541986715 89 89 5 54 2.043e-03 2.016e-03 1.970e-04 1.002e-03
7439 1542004896 1542030234 125 125 6 75 2.999e-03 2.960e-03 1.220e-04 1.044e-03
7440 1542049336 1542077104 163 162 9 96 3.075e-03 3.775e-03 1.660e-04 1.172e-03
7441 1542090535 1542117430 179 169 13 114 1.443e-03 1.459e-03 2.010e-04 9.124e-04
7442 1542135376 1542149279 185 181 15 126 2.068e-03 2.059e-03 2.710e-04 9.953e-04
7443 1542212137 1542233763 192 195 19 144 1.843e-03 1.845e-03 2.390e-04 9.750e-04
7444 1542247020 1542273588 235 238 20 166 2.223e-03 2.231e-03 1.000e-04 1.021e-03
7446 1542309239 1542335749 279 283 22 189 2.249e-03 2.231e-03 9.300e-05 1.012e-03
7447 1542344205 1542344424 279 283 22 189 0.000e+00 1.888e-03 1.600e-04 1.018e-03
7448 1542353526 1542380442 323 327 23 211 2.248e-03 2.660e-03 1.030e-04 1.030e-03
7449 1542389473 1542416067 369 373 25 235 2.261e-03 2.274e-03 1.030e-04 1.050e-03
7450 1542425421 1542451833 412 416 26 257 2.238e-03 2.231e-03 1.010e-04 1.044e-03
7453 1542522977 1542568209 467 472 28 287 2.228e-03 2.274e-03 1.190e-04 1.058e-03
7454 1542584123 1542642529 513 517 30 315 2.231e-03 2.231e-03 8.900e-05 1.121e-02
7456 1542662470 1542687133 555 559 32 340 2.235e-03 2.231e-03 1.020e-04 1.075e-03
7457 1542708236 1542734526 598 602 33 362 2.239e-03 2.231e-03 1.000e-04 1.040e-03
7459 1542755386 1542755976 599 604 33 363 2.720e-03 2.746e-03 5.670e-04 1.016e-03
7460 1542764489 1542781997 633 638 39 379 2.751e-03 2.703e-03 5.700e-04 1.041e-03
7464 1542855492 1542878972 671 677 46 399 2.713e-03 2.703e-03 5.560e-04 1.220e-03
7466 1542910139 1542941206 727 734 55 429 3.522e-03 3.518e-03 5.240e-04 1.087e-03
7467 1542963538 1542992270 791 799 67 458 4.030e-03 4.033e-03 1.205e-03 1.073e-03
7468 1543002191 1543037603 866 875 79 492 5.142e-03 5.148e-03 1.014e-03 1.071e-03
7471 1543066522 1543095796 932 942 91 521 4.995e-03 4.976e-03 9.950e-04 1.102e-03
7472 1543105500 1543133713 998 1009 102 549 5.007e-03 4.976e-03 1.025e-03 1.132e-03
7473 1543142944 1543178229 1074 1085 115 588 6.125e-03 6.092e-03 1.057e-03 1.290e-03
7474 1543194804 1543216131 1133 1144 125 609 5.917e-03 5.834e-03 1.019e-03 1.070e-03
7475 1543251991 1543254308 1143 1147 127 611 6.107e-03 5.577e-03 1.020e-03 1.065e-03
7477 1543271249 1543303184 1210 1214 139 642 5.648e-03 5.663e-03 1.000e-03 1.067e-03
7480 1543360218 1543388616 1278 1285 150 671 5.986e-03 5.963e-03 1.098e-03 1.084e-03
7481 1543399242 1543429605 1351 1360 163 701 5.969e-03 6.006e-03 1.037e-03 1.093e-03
7482 1543443005 1543444906 1360 1368 165 702 5.721e-03 5.577e-03 1.025e-03 1.072e-03
7483 1543455191 1543502081 1409 1415 167 738 4.059e-03 3.947e-03 2.760e-04 1.059e-03
7485 1543523347 1543551628 1466 1471 177 766 4.610e-03 4.547e-03 9.350e-04 1.066e-03
7486 1543567133 1543594487 1522 1527 187 794 4.836e-03 4.633e-03 9.500e-04 1.295e-03
7487 1543604697 1543635280 1588 1594 198 825 5.600e-03 5.620e-03 1.065e-03 1.064e-03
7488 1543646442 1543675593 1655 1660 210 854 5.836e-03 5.749e-03 1.017e-03 1.074e-03
7489 1543685238 1543687919 1668 1673 212 856 5.787e-03 5.792e-03 9.990e-04 1.059e-03
7490 1543696668 1543727362 1736 1741 224 887 5.887e-03 5.834e-03 1.006e-03 1.085e-03
7491 1543736920 1543754334 1788 1794 234 904 5.798e-03 5.749e-03 1.002e-03 1.075e-03
7492 1543762369 1543767055 1797 1802 235 907 2.343e-03 2.231e-03 2.470e-04 7.774e-04
NaN None None None None None None None None None None

The instability table

def LHCInstabilityTable():
    import requests
    response = requests.get('https://lhcinstability.web.cern.ch/lhcinstability/FillSummaries/2018/index.html')
    data = response.text
    aux=[x.split('</a>') for x in data.split('<a href="')]
    aux=aux[1:]
    fill=[int(x[0].split('/')[0]) for x in aux]
    comment=[x[1].split('<!--PLACEHOLDER-->')[0] for x in aux]
    aux=pd.DataFrame({'fill': fill, 'Comment':comment})
    aux=aux.set_index('fill')
    aux.index.name=None
    return aux
LHCInstabilityTable()
Comment
6484 - No instability detected.<br>\n
6492 - No instability detected.<br>\n
6493 - No instability detected.<br>\n
6494 - No instability detected.<br>\n
6495 - No instability detected.<br>\n
6496 \n - Instabilities of probe bunches during the...
6497 - No instability detected.<br>\n
6498 - No instability detected.<br>\n
6499 - No instability detected.<br>\n
6500 - No instability detected.<br>\n
6501 - No instability detected.<br>\n
6502 - No instability detected.<br>\n
6503 - No instability detected.<br>\n
6504 - No instability detected.<br>\n
6505 - No instability detected.<br>\n
6506 - No instability detected.<br>\n
6507 - No instability detected.<br>\n
6508 - No instability detected.<br>\n
6509 - No instability detected.<br>\n
6510 - No instability detected.<br>\n
6511 - No instability detected.<br>\n
6512 - No instability detected.<br>\n
6513 - No instability detected.<br>\n
6514 - No instability detected.<br>\n
6516 - No instability detected.<br>\n
6517 - No instability detected.<br>\n
6518 - No instability detected.<br>\n
6519 - No instability detected.<br>\n
6520 - No instability detected.<br>\n
6521 - No instability detected.<br>\n
... ...
7462 - Loss maps at injection and flat-top with 20...
7463 - No instability detected. Validation cycle w...
7464 - No instability detected.<br>\n
7465 - No coherent instability detected. Dumped by...
7466 - No instability detected.<br>\n
7467 - No instability detected.<br>\n
7468 - No instability detected.<br>\n
7469 - No beam (only probes).<br>\n
7470 - RF resynchronization first. Then beam was r...
7471 - No instability detected.<br>\n
7472 - No instability detected.<br>\n
7473 - No instability detected. Activity in V for ...
7474 - No instability detected.<br>\n
7475 - No instability detected.<br>\n
7477 - No instability detected.<br>\n
7478 - Crystal collimation MD.<br>\n
7479 - Crystal collimation MD.<br>\n
7480 - No instability detected.<br>\n
7481 - no emittance blow-up. Vertical activity at ...
7482 - beams dumped by 10 Hz oscillations 30 minut...
7483 - no emittance blow-up. Vertical activity at ...
7485 - no emittance blow-up. Vertical activity at ...
7486 - no emittance blow-up. Vertical activity at ...
7487 - no emittance blow-up. Vertical activity at ...
7488 - no emittance blow-up. Vertical activity at ...
7489 - no emittance blow-up. Vertical activity at ...
7490 - no emittance blow-up. Vertical activity at ...
7491 - no emittance blow-up. Vertical activity at ...
7492 - no emittance blow-up. Vertical activity at ...
7493 - optics MD. <br>\n

893 rows × 1 columns

Building a fill table

import numpy as np
listDF=[]
myFills=np.arange(6666,6670)
for i in np.unique(myFills):
    aux=importData.LHCFillsByNumber(i)
    myDF=aux[aux['mode']=='FILL'].copy()
    del myDF['mode']
    myDF['modes DF']=[aux[~(aux['mode']=='FILL')]]
    aux=myDF.iloc[0]['modes DF'].groupby('mode').duration.sum()
    aux=pd.DataFrame(aux).transpose()
    aux.columns.name=None
    aux['fill']=i
    aux=aux.set_index('fill')
    aux.index.name=None
    #aux1=importData.LHCCals2pd(['LHC.STATS:B%_NUMBER_BUNCHES'],fillList=[i])
    listDF.append(pd.concat([aux,myDF], axis=1))
fillsDF=pd.concat(listDF,sort=True)

This is the minimal fill DF contatain the specified fill and the relative duration modes.

fillsDF
ADJUST BEAMDUMP FLATTOP INJPHYS INJPROT PRERAMP RAMP RAMPDOWN SETUP SQUEEZE STABLE duration endTime modes DF startTime
6666 00:32:01.453000 00:02:24.449000 00:02:33.174000 00:45:26.077000 00:03:48.286999 00:08:49.490999 00:20:14.409999 00:04:21.061000 NaT 00:10:54.578999 12:43:12.336000 14:53:58.320000 2018-05-11 10:29:36.956000090+00:00 mode start... 2018-05-10 19:35:38.635999918+00:00
6667 NaT NaT NaT 00:43:23.439999 00:10:15.817999 NaT NaT NaT 00:07:09.145999 NaT NaT 01:35:39.635999 2018-05-11 12:05:16.592999935+00:00 mode startT... 2018-05-11 10:29:36.957000017+00:00
6668 NaT NaT NaT 00:10:28.993000 00:04:30.044999 NaT NaT NaT NaT NaT NaT 00:15:09.246999 2018-05-11 12:20:25.841000080+00:00 mode startT... 2018-05-11 12:05:16.594000101+00:00
6669 NaT NaT NaT 00:10:24.634999 00:15:45.363000 NaT NaT NaT NaT NaT NaT 00:29:17.625000 2018-05-11 12:49:43.467000008+00:00 mode startT... 2018-05-11 12:20:25.842000008+00:00
fillsDF[(fillsDF['STABLE']>pd.Timedelta('10h'))]
ADJUST BEAMDUMP FLATTOP INJPHYS INJPROT PRERAMP RAMP RAMPDOWN SETUP SQUEEZE STABLE duration endTime modes DF startTime
6666 00:32:01.453000 00:02:24.449000 00:02:33.174000 00:45:26.077000 00:03:48.286999 00:08:49.490999 00:20:14.409999 00:04:21.061000 NaT 00:10:54.578999 12:43:12.336000 14:53:58.320000 2018-05-11 10:29:36.956000090+00:00 mode start... 2018-05-10 19:35:38.635999918+00:00
importData.LHCFillsAggregation(['LHC.BCTFR.A6R4.B%:BEAM_INTENSITY','LHC.BQM.B%:NO_BUNCHES','LHC.BQM.B1:NO_BUNCHES'],6666, ['INJPHYS'], functionList = np.max, mapInsteadAgg = False)
LHC.BCTFR.A6R4.B1:BEAM_INTENSITY LHC.BCTFR.A6R4.B2:BEAM_INTENSITY LHC.BQM.B1:NO_BUNCHES LHC.BQM.B2:NO_BUNCHES startTime endTime duration
fill mode
6666 INJPHYS 2.833537e+14 2.856917e+14 2556.0 2556.0 2018-05-10 19:39:39.915999889+00:00 2018-05-10 20:25:05.993000031+00:00 00:45:26.077000
importData.LHCFillsAggregation(['LHC.BQM.B%:NO_BUNCHES'], range(6666, 6670),functionList = np.max, mapInsteadAgg = False)
LHC.BQM.B1:NO_BUNCHES LHC.BQM.B2:NO_BUNCHES startTime endTime duration
fill mode
6666 FILL 2556.0 2556.0 2018-05-10 19:35:38.635999918+00:00 2018-05-11 10:29:36.956000090+00:00 14:53:58.320000
6667 FILL 2.0 2.0 2018-05-11 10:29:36.957000017+00:00 2018-05-11 12:05:16.592999935+00:00 01:35:39.635999
6668 FILL 3.0 2.0 2018-05-11 12:05:16.594000101+00:00 2018-05-11 12:20:25.841000080+00:00 00:15:09.246999
6669 FILL 12.0 1.0 2018-05-11 12:20:25.842000008+00:00 2018-05-11 12:49:43.467000008+00:00 00:29:17.625000
cals.search('%NUMBER_%')
['LHC.BSRA.US45.B1/XPocAcquisition#aGXpocNumberAcq',
 'LHC.BSRA.US45.B2/XPocAcquisition#aGXpocNumberAcq',
 'LHC.BSRT.5L4.B2:GATENUMBEROFPULSES',
 'LHC.BSRT.5R4.B1:GATENUMBEROFPULSES',
 'LHC.STATS:B1_NUMBER_BUNCHES',
 'LHC.STATS:B2_NUMBER_BUNCHES',
 'LHC.STATS:NUMBER_COLLISIONS_IP1_5',
 'LHC.STATS:NUMBER_COLLISIONS_IP2',
 'LHC.STATS:NUMBER_COLLISIONS_IP8',
 'LHC.STATS:NUMBER_OF_INJECTIONS',
 'LMFGLO2_STATUS_FLUSHING_NUMBER.POSST',
 'LMFGLO3_STATUS_FLUSHING_NUMBER.POSST',
 'LP.NSRCGEN:NUMBEROFSPARKS',
 'SIMA.10L3.3RM10S:REV_NUMBER_OBSOLETE',
 'SIMA.10L7.7LM08S:REV_NUMBER_OBSOLETE',
 'SIMA.10R3.3RM20S:REV_NUMBER_OBSOLETE',
 'SIMA.10R7.7RM10S:REV_NUMBER_OBSOLETE',
 'SIMA.11L3.3RM09S:REV_NUMBER_OBSOLETE',
 'SIMA.11L7.7LM09S:REV_NUMBER_OBSOLETE',
 'SIMA.11R3.3RM21S:REV_NUMBER_OBSOLETE',
 'SIMA.11R7.7RM11S:REV_NUMBER_OBSOLETE',
 'SIMA.12L3.3RM08S:REV_NUMBER_OBSOLETE',
 'SIMA.12L7.7LM10S:REV_NUMBER_OBSOLETE',
 'SIMA.12R3.3RM22S:REV_NUMBER_OBSOLETE',
 'SIMA.12R7.7RM12S:REV_NUMBER_OBSOLETE',
 'SIMA.13L3.3RM07S:REV_NUMBER_OBSOLETE',
 'SIMA.13L7.7LM11S:REV_NUMBER_OBSOLETE',
 'SIMA.13R3.3RM23S:REV_NUMBER_OBSOLETE',
 'SIMA.13R7.7RM13S:REV_NUMBER_OBSOLETE',
 'SIMA.14L3.3RM06S:REV_NUMBER_OBSOLETE',
 'SIMA.14L7.7LM12S:REV_NUMBER_OBSOLETE',
 'SIMA.14R3.3RM24S:REV_NUMBER_OBSOLETE',
 'SIMA.14R7.7RM14S:REV_NUMBER_OBSOLETE',
 'SIMA.15L3.3RM05S:REV_NUMBER_OBSOLETE',
 'SIMA.15L7.7LM13S:REV_NUMBER_OBSOLETE',
 'SIMA.15R3.3RM25S:REV_NUMBER_OBSOLETE',
 'SIMA.15R7.7RM15S:REV_NUMBER_OBSOLETE',
 'SIMA.16L3.3RM04S:REV_NUMBER_OBSOLETE',
 'SIMA.16L7.7LM14S:REV_NUMBER_OBSOLETE',
 'SIMA.16R3.3RM26S:REV_NUMBER_OBSOLETE',
 'SIMA.16R7.7RM16S:REV_NUMBER_OBSOLETE',
 'SIMA.17L3.3RM03S:REV_NUMBER_OBSOLETE',
 'SIMA.17L7.7LM15S:REV_NUMBER_OBSOLETE',
 'SIMA.17R3.3RM27S:REV_NUMBER_OBSOLETE',
 'SIMA.17R7.7RM17S:REV_NUMBER_OBSOLETE',
 'SIMA.18L7.7LM16S:REV_NUMBER_OBSOLETE',
 'SIMA.18R3.3RM28S:REV_NUMBER_OBSOLETE',
 'SIMA.18R7.7RM18S:REV_NUMBER_OBSOLETE',
 'SIMA.19L7.7LM17S:REV_NUMBER_OBSOLETE',
 'SIMA.19R3.3RM29S:REV_NUMBER_OBSOLETE',
 'SIMA.19R7.7RM19S:REV_NUMBER_OBSOLETE',
 'SIMA.20L7.7LM18S:REV_NUMBER_OBSOLETE',
 'SIMA.20R3.3RM30S:REV_NUMBER_OBSOLETE',
 'SIMA.20R7.7RM20S:REV_NUMBER_OBSOLETE',
 'SIMA.21R7.7RM21S:REV_NUMBER_OBSOLETE',
 'SIMA.4L3.3RM17S:REV_NUMBER_OBSOLETE',
 'SIMA.4L7.7LM01S:REV_NUMBER_OBSOLETE',
 'SIMA.4R7.7RM03S:REV_NUMBER_OBSOLETE',
 'SIMA.5L3.3RM16S:REV_NUMBER_OBSOLETE',
 'SIMA.6L3.3RM15S:REV_NUMBER_OBSOLETE',
 'SIMA.8L3.3RM12S:REV_NUMBER_OBSOLETE',
 'SIMA.8L7.7LM06S:REV_NUMBER_OBSOLETE',
 'SIMA.8R3.3RM18S:REV_NUMBER_OBSOLETE',
 'SIMA.8R7.7RM08S:REV_NUMBER_OBSOLETE',
 'SIMA.9L3.3RM11S:REV_NUMBER_OBSOLETE',
 'SIMA.9L7.7LM07S:REV_NUMBER_OBSOLETE',
 'SIMA.9R3.3RM19S:REV_NUMBER_OBSOLETE',
 'SIMA.9R7.7RM09S:REV_NUMBER_OBSOLETE',
 'SIMA.RE38.3RM31S:REV_NUMBER_OBSOLETE',
 'SIMA.RE38.3RM32S:REV_NUMBER_OBSOLETE',
 'SIMA.RE72.7LM19S:REV_NUMBER_OBSOLETE',
 'SIMA.RE72.7LM20S:REV_NUMBER_OBSOLETE',
 'SIMA.RE78.7RM22S:REV_NUMBER_OBSOLETE',
 'SIMA.RE78.7RM23S:REV_NUMBER_OBSOLETE',
 'SIMA.RR73.7LM05S:REV_NUMBER_OBSOLETE',
 'SIMA.RR77.7RM07S:REV_NUMBER_OBSOLETE',
 'SIMA.UJ33.3RM13S:REV_NUMBER_OBSOLETE',
 'SIMA.UJ33.3RM14S:REV_NUMBER_OBSOLETE',
 'SIMA.UJ76.7RM01S:REV_NUMBER_OBSOLETE',
 'SPS.BWS.41677.V_ROT.APP.IN:BU_NUMBERS',
 'SPS.BWS.41677.V_ROT.APP.OUT:BU_NUMBERS',
 'SPS.BWS.51995.H_ROT.APP.IN:BU_NUMBERS',
 'SPS.BWS.51995.H_ROT.APP.OUT:BU_NUMBERS',
 'SPSQC:LHC_NUMBER_OF_EXECUTED_BATCHES',
 'SPSQC:NUMBER_OF_BUNCHES_FLAT_TOP',
 'SPSQC:NUMBER_OF_INJECTIONS']