Archive for the ‘Various’ Category

June 29 the Feast of The Glorious and First among Apostles Peter and Paul in the Church and What were the names of The twelve Apostles

Tuesday, June 30th, 2020

saint-Glorious-Apostle-Peter-and-Paul-holy-orthodox-icon

Saint Apostle Peter and Paul are the most glorious of all Christ desciples thanks to whom by God's mercy and Grace the nations have received the good news of the Lord Jesus's Christ Crucifix for the sins of all us the sinful people whose evils and unthankfullness is on the way to reach its climax in this days of apostacy where the Church built on top of the Holy Martyrs blood and the Blood and tortures for Christ and the Truth of this two holy man is in one of its biggest temptetation caused by the Coronavirus hysteria exeggerated by the mass-media and purposing to mark a slavery upon human mind and took away the freedoms of man and change the life as we know it.

The Holy Apostles day is the End of the post-Pentecostal fasting which is in the Church from the ancient days of the Church. Rhe date selected being the anniversary of either their death or the translation of their relics.

Fasting is among the 4 main fasts in the One Holy Orthodox Church and the feast in the number of the biggest feasts of the Church. If one reads the historical records for all the places this two simple man was to preach the Gospel he is puzzled and couldn't comprehend how could it be for a simple Roman and a Fisherman to be able to walk through so many lands by boats, ships, through rivers, by carrets, on horses, donkeys, axes, elephants and God knows what kind of other animals typical for the multiple countries and lands this two most holy man has visit. 
It is even more amazing that their frutis of faith planted in the nations are still present today in so many Christians through the world …

The saint Peter and Paul fasting has been set for the reason the Apostles, have fasted immediately after Pentecost the descent of the Holy spirit over The desciples of Christ on the 50th day after the Resurrection of the Lord Jesus Christ. The feast has been setup because the Apostles immerse joy of the Holy Spirit who filled their heart spirits, soul and body made them understood how much dirty and how much in sin they and all the humanity is and how much cleanness is necessery in order for them to start their hard mission of spreading the fact of the Resurrection they have witnessed with their own eyes. A small bracket to open here that Saint Paul never had the  chance to see in Body the Savior just like Saint Paul and the rest of the Apostles, but he was illuminated by The Lord Jesus Christ's appearance to him on the road of the mask when he was on a journey to hunt for christians and put them to court and to death.

Saint Paul was so fierce (by his zillotism) for the Old Testamental Jewish pharisee faith who was based on human interpretation of God's laws, that he was even physically present on the Killing of Saint Archideacon Stephan.
Saint Paul was the one holding the dresses of the Killers of the first Christian martyr St. Stephan thinking that this devilish deed was truthful and pleasant to God. 
But God loved Saint Paul for his Zilotism and his heart aiming to know the Truth and because of that on the Road to Damascus appeared, blinding him from the unbearable light that was emitted by the Lord Jesus Christ who has answered the simple prayer of St. Paul who was honestly looking for the truth. On the Question Saint Paul asked the Unknown bright man who appeared in Glory and surrounded by Angels and Archangels he asked "Who are you Lord"? The reply came, 'I am Jesus, whom you are persecuting'." Acts of the Holy Apostles 9:4-5.

This moment changed Paul forever to make him from a fierce persecutor to a truthful desciple. However as it is said in the Holy Scriptures noone who sees the Lord can't stay alive in the Flesh and perhaps due to that the appearance of Christ left St. Paul to be blind for 3 days until God sent Ananias to heal his eyes by the ordination of Hands in the name of Jesus Christ – again a miracle of God aiming to strengthen the weakness of Faith of Saint Paul and stimulate him to continue on the way of Salvation. Even after the healing of his eyes, later in his eyes by God's providence the eyesight of Paul become weak again and he had to dictate his Apostle letters in the New Testament to his desciples who put it on paper and quite rarely write with his own hand due to his visionary problems. But apperantly the weak physical eye sight doesn't always mean a blindness as with his spiritual eyes the holy apostle was seeing much more than with his physical eyes and one weakness of seeing the physical let him contemplate the eternal.

It is little known fact that saint Paul among with his preaching the Gospel everywhere he went had a profession of making Tents and has worked hardly along with praying day and night, and the sleepless nights of vigil, the tortures by different anti-christians, jews, pagans, philosophers, magicians and others multitude of people who led by his spiritual blindness and passions has done multiple evils and tried in all means to stop Paul to preach the Gospel. But they did not succeeded and we see today the Result as there is rarely a man in the civillized world in all continents who doesn't heard or know about him 20-teen centuries after his martyrdom in the Capital of Roman Empire Rome.

saint-Paul-and-Peter-holy-icon

Saint Peter on the other hand was known for his simplicity and he like all of us was suffering of sickness of weakness of faith, he even rejected to know Christ thrice on the Christ trial, even though he was with Christ for the 3.5 years of Christ's preaching his Salvation Way to the world. But again just like with Paul, God made the miracle of preliminary foretelling him the future, warning him that he is about to reject Christ as his teacher a fact that occured just like prophecised by Christ earlier. Saint Peter seeing that The Lord Jesus Christ as the Son of God knows the future believed him and recover his trust in the Resurrection and with repentance came to believe and await the Resurrection of Christ which by the mercy of God he soon saw with his own eys. Soon after this mercy of God and his preparation with the Eyes of Christ and his desires to follow the will of God for his life led him to completely sacrifice all he had in his remaining earthly life for Christ. Saint Peter "Simon" (Σίμων Simōn in Greek), means stone and he is called that way for the fact he become a stone on which the Church of Christ was build and this stone is present their in the Church and everyone in both Christians and not Christians knows him well.

Saint-Apostle-Paul-and-Peter-embrace

In a dialogue between Jesus and his disciples (Matthew 16:13–19), Jesus asks, "Who do people say that the Son of Man is?" The disciples give various answers. When he asks "Who do you say that I am?", Simon Peter answers, "You are the Messiah, the Son of the living God." Jesus then declares:

Blessed are you, Simon son of Jonah, for this was not revealed to you by flesh and blood, but by my Father in heaven. And I tell you that you are Cephas (Peter) (Petros), and on this rock (petra) I will build my church, and the gates of Hades will not overcome it. I will give you the keys of the kingdom of heaven; whatever you bind on earth will be bound in heaven, and whatever you loose on earth will be loosed in heaven.

Saint Peter is known to have been in Antioch and Corinth and many other lands and is believed to have been the First PopSaint-Apostle-Paul-and-Peter-embrace.jpge of Rome.

Saint-Apostle-Peter-Crucifix-with-head-downed-cross-as-he-said-he-is-unworthy-to-die-as-the-saviour

Early Church tradition says that Peter probably died by crucifixion (with arms outstretched) at the time of the Great Fire of Rome in the year 64. This took place three months after the disastrous fire that destroyed Rome for which the emperor (Nero) wished to blame the Christians. This "dies imperii" (regnal day anniversary) was an important one, exactly ten years after Nero ascended to the throne, and it was "as usual" accompanied by much bloodshed. Traditionally, Roman authorities sentenced him to death by crucifixion. In accordance with the apocryphal Acts of Peter, he was crucified head down. Tradition also locates his burial place where the Basilica of Saint Peter was later built, directly beneath the Basilica's high altar.

On the next day 30th of June the Bulgarian Orthodox Church and some of the other Eastern Orthodox Churches celebrate another great feast The Assembly of the 12 Apostles which honors all the 12 Apostles who were the main building blocks whose preach and martyrdom for Christ put the second stones on the Building of the Church which was based on the main base cornerstone Jesus Christ whom with his holy blood for the Salvation of mankind made the existence of the Ship of Salvation (as the holy fathers) call the Church posslble.

Feast-of-The-Assembly-of-the-Holy-Apostles-and-St-Peter-and-Paul

Below are the Church Troparions and Kontaktions (Praising songs in the Church sang in Holy Liturgy) on 29 of June that every year marks Feast of the Glorious Apostles who enlightened the Universe, I put the songs in Both English and in Cyrillic translated out of Old Bulgarian (Church Slavonic).

Here are the names of the 12 Apostles as we know them by Church Tradition

The 12 disciples of Lord Jesus Christ

1. Peter
2. James
3. John
4. Andrew
5. Bartholomew or Nathanael
6. James, the Lesser or Younger
7. Judas
8. Jude or Thaddeus
9. Matthew or Levi
10. Philip
11. Simon the Zealot
12 . Thomas

 

Troparion — Tone 4

O first-enthroned of the Apostles, / and teachers of the universe, / intercede with the Master of all / to grant peace to the world, / and to our souls great mercy.

Kontakion — Tone 2

O Lord, You have taken to Yourself the steadfast and divinely-inspired heralds, the leaders of Your disciples, / for the enjoyment of Your blessings for and their rest; / for You have accepted their labors and their deaths as above all burnt offerings, / for You alone know the hearts of men.

Kontakion — Tone 2

Today Christ the Rock glorifies with highest honor / the rock of Faith and leader of the Apostles, / together with Paul and the company of the Twelve, / whose memory we celebrate with eagerness of faith, / glorifying Him Who glorified them.

ТРОПАРЬ, ГЛАС 4-Й

Апостолов первопрестольницы и вселенныя учителие, Владыку всех молите мир вселенный даровати и душам нашым велию милость.

Первенствующие из апостолов и Вселенской Церкви учителя, Владыку всех молите мир миру даровать и душам нашим великую милость.

КОНДАК, ГЛАС 2-Й

Твердыя и боговещанныя проповедатели, верх апостолов твоих, Господи, приял еси, в наслаждение благих Твоих и покой: болезни бо онех и смерть приял еси, паче всякаго всеплодия, Едине сведый сердечная.

Непоколебимых и богогласных проповедников, высших из Апостолов Твоих, Господи, Ты принял в наслаждение благ Твоих и покой, ибо страдания их и смерть благоволил принять как жертву, выше всякой жертвы, Единый, ведающий сердца наши.

ВЕЛИЧАНИЕ

Величаем вас, апостоли Христовы Петре и Павле, весь мир ученьми своими просветившия и вся концы ко Христу приведшия.

Прославляем вас, апостолы Христовы Петр и Павел, весь мир своим учением просветивших и приведших ко Христу народы всей земли.

Let by the Prayers of the Holy Apostles Peter and Paul and the Apostle James, John, Andrew, Bartholomew (Nathnael), James (the lesser or Younger), Judas, Jude (Thaddeus), Mathew (Levi), Philip, Simon Zealot and Thomas God have mercy on all Christians in our age and the ages to come till the Second Glorious Coming of Christ and in the Frightening Judgement day.

Make Laptop Sleep on LID (Monitor) close in Linux Debian and Ubuntu systemd Linux

Monday, June 22nd, 2020

make-laptop-auto-sleep-on-lid-close-in-Linux-Ubuntu-Debian-Linux

I need to make my laptop automatically sleep on LID Screen close but it doesn't why?

If have used your laptop for long years with Windows or any Windows user is used to the default beavrior of Windows to automatically sleep the computer on PC close. This default behavior of automatically sleep on LID Close has been Windows standard for many years
and the reason behind that usually laptop is used for mobility and working on a discharging battery so a LID screen close puts the laptop in (SLEEP) BATTERY SUSPEND MODE aiming to make the charged battery last longer. However often for Desktop use in the Office LID close 
trigger of laptop sleep mode is annoying and undesired I've blogged earlier on that issue and how to make laptop not to sleep on LID close on M$ Windows 10 here.

This bahavior was copied and was working in many of the Linux distributions for years however in Debian GNU / Linux and Ubuntu 16.X this feature is often not properly working due to a systemd bug. Of course closing the notebook LID screen without putting
the PC in sleep mode is not a bug but a very useful feature for those who use their laptop as a Desktop machine that is non-stop running, however for most ppl default behavior to auto-suspend the computer on Laptop Monitor close is desired.

Here is how to  force the close of the laptop lid to go to suspend/sleep mode and when open the lid, it wake it up.
 

1. First requirement is to make sure the laptop has installed the package pm-utils, if it is not there install it with:

# apt-get install –yes pm-utils

2. Next we need to edit logind.conf and append 3 variables

# vim /etc/systemd/logind.conf


Normally the file should have a bit of commented informative lines as well as a commented variables that could be enabled like so:

[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
#HandleLidSwitch=suspend
#HandleLidSwitchExternalPower=suspend
#HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#HoldoffTimeoutSec=30s
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RemoveIPC=yes
#InhibitorsMax=8192
#SessionsMax=8192


These entries are usually the files that are used by default as a systemd settings.
Before starting make a copy just you happen to mess systemd.conf, e.g.:

cp -rpf /etc/systemd/logind.conf /etc/systemd/logind.conf_bak


To make the PC LID close active append in the end of file below 3 lines:

HandleSuspendKey=suspend
HandleLidSwitch=suspend
HandleLidSwitchDocked=suspend

systemd-logind-conf-enable-suspend-sleep-on-laptop-lid-screen-close-linux

Save the file and to make systemd daemon reload restart the PC, even though theoretically systemd can be reloaded to digest its new /etc/systemd/logind.conf with:

# systemctl daemon-reexec

3. Assure yourself the Power Management LID setting of the Desktop Graphical User Interface are set to SUSPEND on close


I use MATE Desktop environment as it is simplistic and quite stable fork of GNOME 2.0, anyway depending on the GUI used on the Linux powered laptop e.g. GNOME / KDE Plasma / XFce etc. make sure the respective
 

Control Panel -> Power Management


settings are set to Force the Laptop Screen LID SUSPEND on Close.

Below is how this is done on MATE:

power-management-preferences-when-lid-is-closed-MATE-on-AC-power

power-management-preferences-when-lid-closed-on-battery-suspend

That's all folks, now close your Laptop and enjoy it going to sleep, open it up and get it awaked 🙂 Cheers ! 

What is it like to become a father in the Age of Coronavirus Pandemics – Our baby Dimitar is born

Thursday, June 4th, 2020

After a long 9 months finally on 12.05.2020 12 of May 2020 by God's grace our baby Dimitar was born. He born one day after Saint Cyril and Methodius feast in the Church on the Church Feast of Saint Ephiphanius of Cyprus, Saint German Patriarch of Constantinopol a fierce fighter for the veneration of Holy Icons, Saint martyr Ermogen patriarch of Constantinople (according to new style Calendar) and Saint Basil of Ostrog (in old calendar) . I always loved spring and especially month of May so I'm happy the baby born exactly on this month. For many 2020 broght the coronavirus pandemics brought a lot of pain and surely for us it brought an extra stress with all this mask wearing and super extra precaution measures everywhere and self-isolation but for me 2020 brought me a great joy and a good things in life, after we changed the rented apartment and we moved from Mladost 3 to Geo Milev (a district that is much more fitting my temper), now just 4 months later we have this greatest joy of having a son, something that many people dreamed all their life and suffered. For us it was about 6 years without a baby and the lack of a child in a family seems to extra strain situation. I do suffer and pray for all those people who can't have child and desperately want it and I hope God will bless many with the same joy in the coming years. I have to say having a baby fills up a great hole in the family and brings up new horizons for development of both families and the new born child. Most importantly a new opportunity is there for a new man to get into the kingdom of Heaven know Christ and hopefully end up in eternal blissfulness in Heaven with all the saints by the mercy of God. If you think for a while how all of us some time back in time were also a kid and how our mothers had many sleepless nights and feared for our health and well-being and how from a small baby we become a man who studied excelled in things, failed in others and have the opportunity and rationality to do complex things such as writting this article you get into the conclusion all this is hard to believe mind blowing miracle …

Baby-Dimitar-selected/New-man-born-into-the-world

Right out of Mother's Belly seeing the Light of the World for a first time – First Picture of the Baby before he officially had a name

Many people prayed for the easy birth of my wife as she is already 36 years old and in that years sometimes giving birth is dangerous and often many woman loose babies or are forced to be cut for the baby to be delivered from the belly with Caesarian section cut. Svetlana give a normal birth thanksfully and she delivered the baby for just 3.5 hours after she was accepted in hospital the previous day and doctors did an infusion of oxytocin  (a liquid hormone that doctors use to acccelarate the birth process when the baby was over carried just like it was in our case and in the case of many woman) – Svetlana overcarried it with 5 days.

After a long struggle with my wife on selecting the name, we finally named our new born baby Dimitar  born 49 centimeters / 2980 grams / Dimitar was named in honour of one of the most notorious and loved saints in the Eastern Orthodox world Saint Demitrius of Thessaloniki after a very long struggle to select the name as my wife Svetlana desired to name him Daniil (Daniel), a name which is also beautiful and belongs to the Prophet Daniel and Saint Daniel the Stylite. Svetlana had some weird ideas to name the boy Elijan (Ilia) as well as some other ideas for names like Andrei (Andrew) a very beatiful name belonging to Saint Andrew the Apostle who by the way preached on the Bulgarian Sea Coast according to Church tradition I was against not because the names are bad but because I wanted strongly to follow our well known tradition in Bulgaria to name the first born male boy after the grandfather in that case I wanted to name baby Dimitar firstly in favour of Saint Dimitar (The Myrh Bearer) of Thessaloniki to be the heavinly guide of the boy together with all the other saints under the  Demitrius / Dimitrius name as well as to venerate my father who is a very hard-working and patient parent even over the years with a such a wild child which I am.

Holy Relics of Saint Demetrius the Myrh Bearer in St. Demetrius Basilica in Thessaloniki (Greece)

Saint-Demetrius-the-Myrh-Bearer orthodox holy icon

Saint Demetrius killing Lyaeus the Glariator (depicting the spiritual destroyment of paganism by prayerrs of Saint Demetrius and a remembrance of fact that Christian Nestor killed much powerful Gladiator Lyaeaus who killed thousands of Christians on the Arena before by the all powerful prayers of Saint Demetrius)

I find worthy  to name a few of the other kid's heavinly prayer intercessors this is the well known Russian Saint Dimitrius of Rostov, The bulgarian saint Saint Demitrius of  Besarabia (an ex-territory of Bulgarian Empire) and Saint Dimitrij Donskoy, there is even more saints undet the Demetrius names canonized by the church over the centuries.

The name selection of a boy turned to be much more complicated than I thought and for anyone out there that has to go through the process of awaiting a new born I recommend you to select the name in advance as selecting the name after birth in negotiation with a woman who gave birth is a terrible and hard to bear experience as her hormones are making swing moods every now and then.

/pictures/Baby-Dimitar-selected

Selecting a kid name in the past was quite an interesting process and there was various approaches here in Bulgaria, from naming the kid after a grandfather, grandmother to naming it after a big saint if he is born on a big saint's Church feast day for example if it is born on 6th of May (saint George's day) in Bulgaria it is common to name the kid Georgi or if it is Saint Cyril and Methodius Cyril.
Due to the fact the kid was born near the feast of Saint Apostole Simon the Zealot one of the names I suggested to Svetlana was Simon or Simeon even though that name was not my choice as a compromise that might fit us both. We had some discussion and we both liked the Kiril (Cyril) name, plus 11 of May was Saint Cyril and Methodius but I had an internal tension about it as we didn't have anyone in family called Kiril.

Baby-Dimitar-doctor-checks-heart

Heart works perfect Praise the Lord ! 🙂

Finally my wife stepped back and she agreed to write the name in birth register the name Dimitar so now the kid in his Birth Certificate  is Dimitar Georgiev Georgiev.

Giving birth in Pandemics prevented me to be able to go and see the child until the day he and wife was discharged from Sofia's Maichin Dom University's hospital as clincally healthy.
Please excuse me if I'm turning your attention from the common IT themes Religion and Philosophy which I talk about but I thought putting a few lines for a life changing event as a baby birth is important for me personally to organize things in my head.

/pictures/Baby-Dimitar-selected

The little Big Man

The stress around the baby born is always a big deal both for the mother and the father. But in my case thanks God I was relatively calm. The feelings in the days around birth for the father are quite extreme of course and perhaps this is why many fathers drink till forgetfulness after the baby is born. This however was not the case with me, even though due to the spiritual hardships I have a drinked a couple of beers overall I stayed sober around the birth and right after it before the baby came home.

/pictures/Baby-Dimitar-selected

In front of the Prayer Chapel in Maichin Dom (where yearly the Patriarch of Bulgaria Neofit sanctifies the place with Vodosvet (Sanctification of the Water)

Talking about taking the baby I'm thankful to my dear Friends Angel / Krasimir his wife Irina and Mitko Ivanov, who were the only person to kinda of support me and come for the official dischargement ceremony in hospital. I had to organize a couple of things for the dischargement pay the bills currently in Maichin Dom the overall birth expenses for doctors, midwives, hiring room expenses (for 8 days hospitalization) was lets say normal 1345 LEVA  (~ 700 EURO) much lower price than in other non-government funded hospitals in Sofia like Nadezhda  where it would have been about 2300 LEVA, this is of course higher than social countries of Western Europe like Germany where a normal state funded birth would cost something like ~ 350 – 400 EUR but still very cheap if Compared to United Stateswhere a good orchestrated birth costs something like 25 to 30 000 USD.
As I heard from wife the birth experience she got was of course harsh but this is normal for the first baby where the levels of stress and uncertainty is absolutely unbearable for the your unexperienced parturient mother.

I have to express my sincere thankfulness to the great Head Doctor Miss Ivet Raicheva thanks to whom my wife succeded in normal birth and we have a healthy baby.as well Doctor Nikolay Gerdzhikov from Hospital Second Baby Specialized Hospital Sheinovo who  break off the amniotic fluids baloon of my wife to accelarate the overcarried baby timely birth, as well as all the pregnancy tracking doctors of UMBAL Nadezhda (A Hispital for Woman Health).

Just like I thank warmly to all the people who have given us baby clothes, baby car chairs, subtrates, carriage cangoroos and all kind of baby toys and equipment useful in raising the baby as well as all the friends who helped with advices during the pregnancy and many hardships in this 9 months before baby come to earth and after that. This are Mitko Paskalev, Mitko Ivanov / Anastasia, Krasimir, Hristina, Father Stoyan and his wife Yanna, our godfather Familiy Galin and Andrea, uncle Emilian, Vasil Kolev, Father Flavian and all others who helped us with warm prayers and good words during the hardships of pregnancy during the Coronacrisis.

Due to the Covid, every time I had to go to the hospital to bring my wife food, pampers, fruits etc. was only possible to be delivered by a medicine personal (with a small treatment fee) as entrance of externals like me was not possible.

I did not have the chance to go inside the hospital's 12th floor to pick up my wife with the baby due to the COVID-19 Virus, hospital entrance was only allowed to the parter stage and only after they check your temperature with an electronic wireless gun-like thermometer headed right in your head …
I had to then wait with the few bouquet of flowers, chocolate candys and alcohol to hand in to the main degenerating doctor which in our case was Ivet Raicheva, I have to kindly thank this professional woman for doing all the best for my wife in assisting her in birth and succeeding in a normal birth process which in our age is quite rare about at least 80% of woman give birth with a C-Section.

Baby-with-Angel-Krasimir-Irina-and-Mitko-Ivanov

Friends and Brothers / Sisters from the Church Angel, Krasi, Irina and Mitko Ivanov

/pictures/Baby-Dimitar-selected/Baby-Krasi-Irina-and-Sveta

Krasimir and Irina

/pictures/Baby-Dimitar-selected

In front of Maichin Dom Me seeing my boy for a first time !

After Svetlana was accompanied in the entrance stage with a medicine worker, we made the standard few remembrance pictures on the floor and infront the hospital and on a Volkswagen Taxi headed home with the baby being in fear for the baby in every car bump.

aking-the-baby-home

The great joy of blessing to be with your Son for a first time

Once Dimitar was already home we rejoiced and placed him in his already prepared baby crib and left home wife for 40 minutes together with the baby and went out to for a quick treat for friends who were so kind to come for the baby.
The routine afterwards is expected as to every new born, a lot of breast feeding for wife, adaptated milk sometimes, changing pampers, baby bathing every day, swinging, singing songs to calm him down when he songs etc.

Baby-Dimitar-selected/Baby-Dimitar-Svetlana1

The responsibilities for the father of course suddenly rise as you have to be a products supporter as your wife is quite weak over the 40 days after birth, you have to clean, buy food or prepare something to eat, prepare her a breastfeeding teas, confort her and calm her. But the overall it is clear that the woman becomes much more stable version of herself after the birth she starts thinking more to the ground and dream less in fantasies as the baby helps her better see the reality and learn to sacrifice more.

Georgi-Baby-Dimitar

Let God bless and protect Dimitar by the prayers of the Holy Virgin Mary Theotokos and All Sains and help him in all the hardships from the cradle to a fully grown and wise man that he'll become one day by God's mercy!

Christ is Risen Eastern Orthodox Resurrection Paschal Greeting in Different Languages

Friday, April 24th, 2020

Resurrection-of-Christ-holy-orthodox-icon-Voskresenie-Hristovo

Happy Easter to All Orthodox Christians worldwide ! 
We are the the bright week –  this is the week after The Feast of Feasts Resurrection of Christ. This year in 2020, we Orthodox Christians celebrated this feast on 19th, 20th and 21st of April. The celebrations of the Feast of Christians and the Paschal joy continues for the whole week, so I found some time to quickly blog to share the Joy of the Resurrection of the Savior Jesus Christ who has freed all People from the Fear of the Death by Manifestating Death has been overtaken by Life Eternally.
Earlier years, I've blogged shortly on the Christ is risen in many languages. But this time I decided to extend my previous blog by adding some more details on which are the Member Churches consisting the Christ body of Holy Orthodoxy, What is the Creed of Faith (Symbol of Faith) difference Between Orthodox Christians and Roman Catholics and why we're not catholics and do celebrate Easter on a different date from Roman Catholics. Finally I will post the Paschal Greeting translated to as many languages I could find.

In the Eastern Orthodox Christian world which is the True Church of Christ consists of 15 National Churches each traced back to the Holy Apostles of Christ, each of Churches is in eucharistic Communion with the other. 

Canonical Orthodox Christ Churches as of year 2020 are the following:

1. Orthodox Church of Constantinople
2. Orthodox Church of Alexandria
3. Orthodox Church of Antiochia
4. Jerusalem Patriarchal Orthodox Church
5. Bulgarian Orthodox Church
6. Georgian Orthodox Church
7. Serbian Orthodox Church
8. Russian Orthodox Church
9. Romanian Orthodox Church
10. Orthodox Church of Cyprus (archibishopship)
11. Greek Orthodox Church
12. Albanian Orthodox Church
13. Polish Orthodox Church
14. Orthodox Church of Chech Lands and Slovakia
15. American Orthodox Church


Historically Christ Church was one before the Great Schism that was perhaps the greatest tragedy of mankind after Christ's Crucifix it occured in 1054 A.D. About this saddening events, the manuscripts and contemporary saints says with this terrible event, the whole world shaked its basis. The result of the Bulla brought by Pope's messangers in Hagia Sophia Cathedral in Constantinople in the Holy Alter of the Church putting in document of official schism and the Church of the east condeming the Western Church Cuhrch action headed by the pope due to the false Creed of faith inclusion legallized by the pope with the so called 'Filioque' word formula that changed the original agreement of Church fathers decisions on the First Ecumenical Council of Constantinople in 381 A.D. (which by the way puts Anathemas on anyone who dares to change the Creed of Faith as well change by the Popes in the well known ancient Baptism Formulas like oilment (receving the sign of the holy spirit during baptism).

The historical number of Orthodox Churche dioceses were much more numerous but with the time and the hardships this are the only ones that left as official Churches, many dioceses were destroyed by the Muslims Conquests and Roman Catholics orders like the Jesuits whose fight against orthodoxy has been severe in their attempt to make the whole world to turn to the pope, this is very well known by the many remains of Uniates around Europe, especially in nowadays Ukraine. There is a lot of nations like Chechz and Hungarians whose for many centuries confessed orthodoxy but due to the economic relations with the West and the converion of their rulers (princes / Kings) etc. to Roman Catholicism has gradually converted their Eastern Churches to Roman Catholics.

The origional Nicean Creed (Nicea-Constantinople) creed of faith reads as this:

Nicean Creed of Faith ( Agreed on 381 Anno Dommini in Emperor Constantine City of Byzantine Empire Constantinople)

We believe in one God, the Father Almighty, Maker of heaven and earth, and of all things visible and invisible.

And in one Lord Jesus Christ, the only-begotten Son of God, begotten of the Father before all worlds (æons), Light of Light, very God of very God, begotten, not made, consubstantial with the Father;

by whom all things were made;

who for us men, and for our salvation, came down from heaven, and was incarnate by the Holy Ghost and of the Virgin Mary, and was made man;

he was crucified for us under Pontius Pilate, and suffered, and was buried, and the third day he rose again, according to the Scriptures, and ascended into heaven, and sitteth on the right hand of the Father;

from thence he shall come again, with glory, to judge the quick and the dead. ;

whose kingdom shall have no end.

* And in the Holy Ghost, the Lord and Giver of life, who proceedeth from the Father, who with the Father and the Son together is worshiped and glorified, who spake by the prophets.

In one Holy Catholic and apostolic Church; we acknowledge one baptism for the remission of sins; we look for the resurrection of the dead, and the life of the world to come. Amen.

The Western Church head Bishop the Pope and local priests due to some historical regions of Spain and other parts of Western Europe's aim to fight heresies included the word Latin word Filioque in above translated text (Word which is translated as "And from the Son") in above starred line 'And in te Holy Ghost, The Lord Giver of Life who proceedeth from the Father' become 'And in the Holy Ghost, The Lord Giver of Life who proceedeth from the Father (Filioque) = and from the Son.' this was acceptable for the Eastern Churches until the moment when this Confession of Faith has been legalized for the Whole Western Church with a decree so called pope 'Bulla' with which it become the official confession of faith for the whole Catholic Church. The Eastern Church of course was following the accepted Canon rules from the first Ecumenical Council in 381 A.D. and rejected to accept the definition of the Pope at first in the Face of Saint Patriarch Photios I of Constantinople (year 810 –  893) and become official in 1054 by the rule of Pope Leo whose legates tried to claim Headship of the Pope over the whole Church and questioned the title of the Constantinople Ecumenical Patriarch Michael I Cerularius.
Along with the chages of the Creed of Faith the West, the years during centuries VII and IX centuries has already put a lot of differences in the East and West Church along doctrinal, theological, linguistic, political, and geographical lines so the split was a reflection of all this. The Latin Church was much more power hungry and more progressive for its time and authoritarian, trying to combine the Worldly power with the Spiritual one given by the line of Apostoles from Christ Ceasaris-Papism, where the Eastern Church was governed in the ancient model of the Worldly power in face of Eastern Roman empire Emperor and the Patriarch who was a governor of the Spiritual power. The schism was worsened also by the many Latins raids in the Eastern Empire Christian brothers and the sacking of Constantinople in 8-13 April year 1204. Of course both Wester and Eastern Roman Empire had an appetite for a conquest over the other and often this has lead the secular rulers on both sides to try to manipulate activities of the spiritual leaders of both to work for their interests, but the schism would never occur if the spiritual establishment of the Church which are the Holy Canons (decision of the Ecumenical Councils) were not breached by the Western Church.
One of this breaches of the Ancient canons is the Celebration of Eastern Pascha which says the Christian Pascha should never coincide with Jewish Pascha. However in the Western Church this rule was breached and nowadays The Eastern (The Day of the Resurrection of Christ) in the Roman Catholic Church (Western Church) coincides most of the years with Jewish Pascha (both Roman Catholics and the executors of Christ who never accepted him the Jews celebrate together … a sad fact).

Nowadays most of the Ancient Churches of the East together with the Eastern Orthodox Churches, who are confessing the Faith of Christ such as it was handed by the Saint Fathers has a very specific ancient way of confession of faith similar to the Creed of Faith which was a very common short ancient way to confess the faith when two Christians met it is perhaps originating from the times of the Heresies in the 1st century right after the Christ Crucifix, when the pupil of Christ used it to confirm the Glorious and unexplainable Miracle of the Resurrection of the Lord Jesus Christ from the Death in Real Body in the 3rd day from the Grave in the Cave where his body was buried.

Resurrection-of-Christ-Anastasis-Greek-orthodox-icon

The Greeting Formula is the well known in the Eastern Orthodox Churches such as in Bulgaria / Greece / Russia / Serbia etc. Christ is Risen.
On every easter Almost everyone in the Orthodox Christian Countires greats everyone else both in homes on the street at work or anywhere relatives friends and even unfamiliar people who has to do business deeds with the immersely joyful greeting.

ХРИСТОС ВОСКРЕСЕ / CHRIST IS RISEN !!!!! !!!

Then the greeted Person answers back

ВОЙСТИНУ ВОСКРЕСЕ / TRULY HE IS RISEN (INDEED HE IS RISEN) !!!!!!!!

In the Orthodox Churches, believers do greet themselves with this heartful joyful greeting for the whole 40 days after the Feast of Resurrection of Christ.

In Russia, Ukrain, Belarus and the surrounding Slavonic lands there is this tradition that the greeting is repeated 3 times as an interaction between person A and person B, for example.

Person A (3 times) greets:
ХРИСТОС ВОСКРЕСЕ = CHRIST IS RISEN !!!
Person B (3 times) answers:
ВОЙСТИНУ ВОСКРЕСЕ = TRULY HE IS RISEN !!!

Resurrection-of-Christ-Velikden-orthodox-holy-icon

Below is a good list with Paschal Resurrection Greeting in multiple languages, for those who has curious polyglot minds who want to learn few words in different languages.

Indo-European languages

Greek: Χριστὸς ἀνέστη! Ἀληθῶς ἀνέστη! (Khristós anésti! Alithós anésti!)

Voskresenie-Gospoda-Nashego-Iisusa-Hrista-Mosaic

Slavic languages

Church Slavonic: Хрїсто́съ воскре́се! Вои́стинꙋ воскре́се! (Xristósŭ voskrése! Voístinu voskrése!)

Belarusian: Хрыстос уваскрос! Сапраўды ўваскрос! (Chrystos uvaskros! Sapraŭdy ŭvaskros!)

Bulgarian: Христос воскресе! Воистину воскресе! (Khristos voskrese! Voistinu voskrese!), as if in Church Slavonic; Христос възкресе! Наистина възкресе! (Khristos vâzkrese! Naistina vâzkrese!) in Modern Bulgarian

Croatian: Krist uskrsnu! Uistinu uskrsnu!

: Kristus vstal z mrtvých! Vpravdě vstal z mrtvých!

Macedonian: Христос воскресе! Навистина воскресе! (Hristos voskrese! Navistina voskrese!), traditional; or Христос воскресна! Навистина воскресна! (Hristos voskresna! Navistina voskresna!)

Polish: Chrystus zmartwychwstał! Prawdziwie zmartwychwstał!

Russian: Христос воскрес(-е)! Воистину воскрес(-е)! (Khristos voskres(-е)! Voistinu voskres(-е)!) (the version with -e is in Church Slavonic, one without it is in modern Russian; both are widely used)

Rusyn: Хрістос воскрес! Воістину воскрес! (Hristos voskres! Voistynu voskres!)

Serbian: Христос васкрсе! Ваистину васкрсе! (Hristos vaskrse! Vaistinu vaskrse!) or Христос воскресе! Ваистину воскресе! (Hristos voskrese! Vaistinu voskrese!)

Slovak: Kristus vstal z mŕtvych! Skutočne vstal (z mŕtvych)! (though the Church Slavonic version is more often used)

Slovene: Kristus je vstal! Zares je vstal!

Ukrainian: Христос воскрес! Воістину воскрес! (Khrystos voskres! Voistynu voskres!)

Tosk Albanian: Krishti u ngjall! Vërtet u ngjall!

Armenian

Western Armenian: Քրիստոս յարեա՜ւ ի մեռելոց: Օրհնեա՜լ է Յարութիւնն Քրիստոսի: (Krisdos haryav i merelotz! Orhnyal e Haroutyunen Krisdosi!)

eastern dialect, Քրիստոս հարյա՜վ ի մեռելոց: Օրհնյա՜լ է Հարությունը Քրիստոսի: (Khristos haryav i merelotz! Orhnyal e Harouthyoune Khristosi!); literally "Christ is risen! Blessed is the resurrection of Christ!")
 

Christus-Resurrexit-Latin-icon-Christ_Resurrection

Germanic languages

Anglic languages

Scots: Christ has ryssyn! Hech aye, he his ain sel!

English: Christ is risen! He is risen indeed! Or Christ is risen! Truly, he is risen!

Old English: Crist is ārisen! Hē is sōþlīċe ārisen!

Middle English: Crist is arisen! Arisen he sothe!

Danish: Kristus er opstanden! Sandelig Han er Opstanden!

West Frisian: Kristus is opstien! Wis is er opstien!

German: Christus ist auferstanden! Er ist wahrhaft auferstanden! or Der Herr ist auferstanden! Er ist wahrhaftig auferstanden!

Icelandic: Kristur er upprisinn! Hann er sannarlega upprisinn!

Faroese: Kristus er upprisin! Hann er sanniliga upprisin!

Low Franconian languages

Dutch: Christus is opgestaan! Hij is waarlijk opgestaan! (Netherlands) or Christus is verrezen! Hij is waarlijk verrezen! (Belgium)

Afrikaans: Christus het opgestaan! Hy het waarlik opgestaan!

Norwegian

Bokmål: Kristus er oppstanden! Han er sannelig oppstanden!

Nynorsk: Kristus er oppstaden! Han er sanneleg oppstaden!

Swedish: Kristus är uppstånden! Han är sannerligen uppstånden!

Italic languages

Latin: Christus resurrexit! Resurrexit vere!

Romance languages

Aromanian: Hristolu anyie! Di alihea anyie!

Resurrection-of-Christ-Coptic-christian-icon

Catalan: Crist ha ressuscitat! Veritablement ha ressuscitat!

French: Le Christ est ressuscité ! En vérité il est ressuscité! Or Le Christ est ressuscité ! Vraiment il est ressuscité !

Galician: Cristo resucitou! De verdade resucitou!

Italian: Cristo è risorto! È veramente risorto!

Portuguese: Cristo ressuscitou! Em verdade ressuscitou! or Cristo ressuscitou! Ressuscitou verdadeiramente!

Arpitan: Lo Crist es ressuscitat! En veritat es ressuscitat!

Romanian: Hristos a înviat! Adevărat a înviat!

Romansh: Cristo es rinaschieu! In varded, el es rinaschieu!

Sardinian: Cristu est resuscitadu! Aberu est resuscitadu!

Sicilian: Cristu arrivisciutu esti! Pibbiru arrivisciutu esti!

Spanish: ¡Cristo resucitó! ¡En verdad resucitó!

Walloon: Li Crist a raviké! Il a raviké podbon!

Baltic languages

Latvian: Kristus (ir) augšāmcēlies! Patiesi (viņš ir) augšāmcēlies!

Lithuanian: Kristus prisikėlė! Tikrai prisikėlė!
 

Celtic languages

Goidelic languages

Old Irish: Asréracht Críst! Asréracht Hé-som co dearb!

Irish: Tá Críost éirithe! Go deimhin, tá sé éirithe!

Manx: Taw Creest Ereen! Taw Shay Ereen Guhdyne!

Scottish Gaelic: Tha Crìosd air èiridh! Gu dearbh, tha e air èiridh!

Brythonic languages

Breton:Dassoret eo Krist! E wirionez dassoret eo!

Cornish: Thew Creest dassorez! En weer thewa dassorez!

Welsh: Atgyfododd Crist! Yn wir atgyfododd!

Indo-Iranian languages

Ossetian:Чырысти райгас! Æцæгæй райгас! Or бæлвырд райгас! (Ḱyrysti rajgas! Æcægæj rajgas or bælvyrd rajgas!)

Persian: مسیح برخاسته است! به راستی برخاسته است!‎ (Masih barkhaste ast! Be rasti barkhaste ast!)

Hindi: येसु मसीह ज़िन्दा हो गया है! हाँ यक़ीनन, वोह ज़िन्दा हो गय یسوع مسیح زندہ ہو گیا ہے! ہاں یقیناً، وہ زندہ ہو گیا ہے!‎ (Yesu Masīh zindā ho gayā hai! Hā̃ yaqīnan, voh zindā ho gayā hai!)

Marathi: Yeshu Khrist uthla ahe! Kharokhar uthla ahe!

Abkhazian: Kyrsa Dybzaheit! Itzzabyrgny Dybzaheit!
 

Jesus-Christ-Resurrected-arabic-coptic-icon

Afro-Asiatic languages

 

Semitic languages

 

Standard Arabic: المسيح قام! حقا قام!‎ (al-Masīḥ qām! Ḥaqqan qām!) or المسيح قام! بالحقيقة قام! (al-Masīḥ qām! Bi-l-ḥaqīqati qām!)

Aramaic languages

 

Classical Syriac: ܡܫܝܚܐ ܩܡ! ܫܪܝܪܐܝܬ ܩܡ!‎ (Mshiḥa qām! sharīrāīth qām! or Mshiḥo Qom! Shariroith Qom!)

Assyrian Neo-Aramaic: ܡܫܝܚܐ ܩܡܠܗ! ܒܗܩܘܬܐ ܩܡܠܗ!‎ (Mshikha qimlih! bhāqota qimlih!)

Turoyo: ܡܫܝܚܐ ܩܝܡ! ܫܪܥܪܐܝܬ ܩܝܡ!‎ (Mshiḥo qāyem! Shariroith qāyem!)

East African languages

Tigrinya: Christos tensiou! Bahake tensiou!

Amharic: Kristos Tenestwal! Bergit Tenestwal!

Hebrew: המשיח קם! באמת קם!‎ (Hameshiach qam! Be'emet qam!)

Maltese: Kristu qam! Huwa qam tassew! or Kristu qam mill-mewt! Huwa qam tassew!

Egyptian

Coptic: (Pi'Christos aftonf! Khen oumetmi aftonf!)

Judeo-Berber: Lmasih yahye-d ger lmeytin! Stidet yahye-d ger lmeytin!


Dravidian languages

Tamil: கிறிஸ்து உயிர்த்தெழுந்தார், மெய்யாகவே அவர் உயிர்த்தெழுந்தார்.

Malayalam: ക്രിസ്തു ഉയിര്ത്തെഴുന്നേറ്റു! തീര്ച്ചയായും ഉയിര്ത്തെഴുന്നേറ്റു! (Christu uyirthezhunnettu! Theerchayayum uyirthezhunnettu!)
 

Eskimo–Aleut languages

Aleut: Kristusaaq Aglagikuk! Angangulakan Aglagikuk!

Pacific Gulf Yupik: Kristusaq ungwektaq! Pichinuq ungwektaq!

Central Yupik: Kristuussaaq unguirtuq! Ilumun unguirtuq!

Mayan languages

Tzotzil: Icha'kuxi Kajvaltik Kristo! Ta melel icha'kuxi!

Tzeltal: Cha'kuxaj Kajwaltik Kristo! Ta melel cha'kuxaj!

Christ-resurrection-Anastasis

 

Austronesian languages

Malayo-Polynesian

Batak: Tuhan nunga hehe! Tutu do ibana hehe!

Carolinian: Lios a melau sefal! Meipung, a mahan sefal!

Cebuano: Nabanhaw Si Kristo! Nabanhaw gayud!

Waray: Hi Kristo nabanwaw! Matuod nga Hiya nabanhaw!

Chamorro: La'la'i i Kristo! Magahet na luma'la' i Kristo!

Fijian: Na Karisito tucake tale! Io sa tucake tale!

Filipino: Nabuhay muli Si Kristo! Nabuhay talaga!

Hawaiian: Ua ala hou ʻo Kristo! Ua ala ʻiʻo nō ʻo Ia!

Indonesian: Kristus telah bangkit! Dia benar-benar telah bangkit!

Kapampangan: Y Kristû sinûbli yáng mèbié! Sinûbli ya pin mèbié!

Malagasy: Nitsangana tamin'ny maty i Kristy! Nitsangana marina tokoa izy!

Cook Islands Māori: Kuo toetu’u ‘ae Eiki! ‘Io kuo toetu’u mo’oni!

 Austroasiatic languages: Mon-Khmer

: Preah Christ mean preah choan rous leong vinh! trung mean preah choan rous leong vinh men!

 Vietnamese

: Chúa Ki-tô đã sống lại! Ngài đã sống lại thật!

 Thai

Thai: พระคริสต์เป็นขึ้นจากความตาย! or พระคริสต์ทรงกลับคืนพระชนม์ชีพ!

Basque

Basque: Cristo Berbiztua! Benetan Berbiztua!

Japanese

Japanese: ハリストス復活!実に復活! (Harisutosu fukkatsu! Jitsu ni fukkatsu!)

 Korean

Korean 그리스도 부활하셨네! 참으로 부활하셨네! (Geuriseudo buhwalhasyeonne! Chameuro buhwalhasyeonne!)

 Na-Dené languages

Athabaskan languages

Navajo: Christ daaztsą́ą́dę́ę́ʼ náádiidzáá! Tʼáá aaníí daaztsą́ą́dę́ę́ʼ náádiidzáá!

Tlingit: Xristos Kuxwoo-digoot! Xegaa-kux Kuxwoo-digoot!

Niger–Congo languages

: Kristo Ajukkide! Kweli Ajukkide!

Swahili: Kristo Amefufuka! Amefufuka kweli kweli!

Gikuyu: Kristo ni muriuku! Ni muriuku nema!

Quechuan languages

Quechua: Cristo causarimpunña! Ciertopuni causarimpunña!

Mongolic languages

Classical Mongolian: Есүс дахин амилсан, Тэр үнэхээр амилсан! (Yesus dahin amilsan, ter uneheer amilsan)

Turkic languages

Turkish: Mesih dirildi! Hakikaten dirildi!

Uyghur: ‫ئەيسا تىرىلدى! ھەقىقەتىنلا تىرىلدى!‬‎ (Əysa tirildi! Ⱨəⱪiⱪətinla tirildi!)

Azerbaijani: Məsih dirildi! Həqiqətən dirildi!

Chuvash: Христос чĕрĕлнĕ! Чăн чĕрĕлнĕ! (Hristos čĕrĕlnĕ! Čyn čĕrĕlnĕ!)

Khakas: Христос тірілді! Сыннаң тірілді! (Hristos tíríldí! Sınnañ tíríldí!)

Uzbek: Масих тирилди! Хақиқатдан тирилди! (Masih tirildi! Haqiqatdan tirildi!)

Sino-Tibetan languages

Chinese: 基督復活了!他確實復活了! (Jīdū fùhuó-le! Tā quèshí fùhuó-le!) or 耶穌復活了,真的他復活了! (Yēsū fùhuó-le, Zhēnde tā fùhuó-le!)

Uralic languages

Estonian: Kristus on üles tõusnud! Tõesti on üles tõusnud!

Finnish: Kristus nousi kuolleista! Totisesti nousi!

Hungarian: Krisztus feltámadt! Valóban feltámadt!

Karelian: Hristos nouzi kuollielois! Tovessah nouzi!

Glorious-Resurrection-of-Jesus-Christ-icon

Constructed languages

International auxiliary languages

Esperanto: Kristo leviĝis! Vere Li leviĝis!

Ido: Kristo riviveskabas! Ya Il rivivesakabas!

Interlingua: Christo ha resurgite! Vermente ille ha resurgite! or Christo ha resurrecte! Vermente ille ha resurrecte!

Quenya: (Hristo Ortane! Anwave Ortanes!)

Klingon: Hu'ta' QISt! Hu'bejta'!

 

Find when cron.daily cron.weekly and cron.monthly run on Redhat / CentOS / Debian Linux and systemd-timers

Wednesday, March 25th, 2020

Find-when-cron.daily-cron.monthly-cron.weekly-run-on-Redhat-CentOS-Debian-SuSE-SLES-Linux-cron-logo

The problem – Apache restart at random times


I've noticed today something that is occuring for quite some time but was out of my scope for quite long as I'm not directly involved in our Alert monitoring at my daily job as sys admin. Interestingly an Apache HTTPD webserver is triggering alarm twice a day for a short downtime that lasts for 9 seconds.

I've decided to investigate what is triggering WebServer restart in such random time and investigated on the system for any background running scripts as well as reviewed the system logs. As I couldn't find nothing there the only logical place to check was cron jobs.
The usual
 

crontab -u root -l


Had no configured cron jobbed scripts so I digged further to check whether there isn't cron jobs records for a script that is triggering the reload of Apache in /etc/crontab /var/spool/cron/root and /var/spool/cron/httpd.
Nothing was found there and hence as there was no anacron service running but /usr/sbin/crond the other expected place to look up for a trigger even was /etc/cron*

1. Configured default cron execution times, every day, every hour every month

# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 feb 27 10:54 /etc/cron.d/
drwxr-xr-x 2 root root 4096 dec 27 10:55 /etc/cron.daily/
drwxr-xr-x 2 root root 4096 dec  7 23:04 /etc/cron.hourly/
drwxr-xr-x 2 root root 4096 dec  7 23:04 /etc/cron.monthly/
drwxr-xr-x 2 root root 4096 dec  7 23:04 /etc/cron.weekly/

After a look up to each of above directories, finally I found the very expected logrorate shell script set to execute from /etc/cron.daily/logrotate and inside it I've found after the log files were set to be gzipped and moved to execute WebServer restart with:

systemctl reload httpd 

My first reaction was to ponder seriously why the script is invoking systemctl reload httpd instead of the good oldschool

apachectl -k graceful

But it seems on Redhat and CentOS since RHEL / CentOS version 6.X onwards systemctl reload httpd is supposed to be identical and a substitute for apachectl -k graceful.
Okay the craziness of innovation continued as obviously the reload was causing a Downtime to be visible in the Zabbix HTTPD port Monitoring graph …
Now as the problem was identified the other logical question poped up how to find out what is the exact timing scheduled to run the script in that unusual random times each time ??
 

2. Find out cron scripts timing Redhat / CentOS / Fedora / SLES

/etc/cron.{daily,monthly,weekly} placed scripts's execution method has changed over the years, causing a chaos just like many Linux standard things we know due to the inclusion of systemd and some other additional weird OS design changes. The result is the result explained above scripts are running at a strange unexpeted times … one thing that was intruduced was anacron – which is also executing commands periodically with a different preset frequency. However it is considered more thrustworhty by crond daemon, because anacron does not assume the machine is continuosly running and if the machine is down due to a shutdown or a failure (if it is a Virtual Machine) or simply a crond dies out, some cronjob necessery for overall set environment or application might not run, what anacron guarantees is even though that and even if crond is in unworking defunct state, the preset scheduled scripts will still be served.
anacron's default file location is in /etc/anacrontab.

A standard /etc/anacrontab looks like so:
 

[root@centos ~]:# cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron
 
# See anacron(8) and anacrontab(5) for details.
 
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
 
#period in days   delay in minutes   job-identifier   command
1    5    cron.daily        nice run-parts /etc/cron.daily
7    25    cron.weekly        nice run-parts /etc/cron.weekly
@monthly 45    cron.monthly        nice run-parts /etc/cron.monthly

START_HOURS_RANGE : The START_HOURS_RANGE variable sets the time frame, when the job could started. 
The jobs will start during the 3-22 (3AM-10PM) hours only.

  • cron.daily will run at 3:05 (After Midnight) A.M. i.e. run once a day at 3:05AM.
  • cron.weekly will run at 3:25 AM i.e. run once a week at 3:25AM.
  • cron.monthly will run at 3:45 AM i.e. run once a month at 3:45AM.

If the RANDOM_DELAY env var. is set, a random value between 0 and RANDOM_DELAY minutes will be added to the start up delay of anacron served jobs. 
For instance RANDOM_DELAY equels 45 would therefore add, randomly, between 0 and 45 minutes to the user defined delay. 

Delay will be 5 minutes + RANDOM_DELAY for cron.daily for above cron.daily, cron.weekly, cron.monthly config records, i.e. 05:01 + 0-45 minutes

A full detailed explanation on automating system tasks on Redhat Enterprise Linux is worthy reading here.

!!! Note !!! that listed jobs will be running in queue. After one finish, then next will start.
 

3. SuSE Enterprise Linux cron jobs not running at desired times why?


in SuSE it is much more complicated to have a right timing for standard default cron jobs that comes preinstalled with a service 

In older SLES release /etc/crontab looked like so:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly


As time of writting article it looks like:

SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root
#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * *   root  test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1


This runs any scripts placed in /etc/cron.{hourly, daily, weekly, monthly} but it may not run them when you expect them to run. 
/usr/lib/cron/run-crons compares the current time to the /var/spool/cron/lastrun/cron.{time} file to determine if those jobs need to be run.

For hourly, it checks if the current time is greater than (or exactly) 60 minutes past the timestamp of the /var/spool/cron/lastrun/cron.hourly file.

For weekly, it checks if the current time is greater than (or exactly) 10080 minutes past the timestamp of the /var/spool/cron/lastrun/cron.weekly file.

Monthly uses a caclucation to check the time difference, but is the same type of check to see if it has been one month after the last run.

Daily has a couple variations available – By default it checks if it is more than or exactly 1440 minutes since lastrun.
If DAILY_TIME is set in the /etc/sysconfig/cron file (again a suse specific innovation), then that is the time (within 15minutes) when daily will run.

For systems that are powered off at DAILY_TIME, daily tasks will run at the DAILY_TIME, unless it has been more than x days, if it is, they run at the next running of run-crons. (default 7days, can set shorter time in /etc/sysconfig/cron.)
Because of these changes, the first time you place a job in one of the /etc/cron.{time} directories, it will run the next time run-crons runs, which is at every 15mins (xx:00, xx:15, xx:30, xx:45) and that time will be the lastrun, and become the normal schedule for future runs. Note that there is the potential that your schedules will begin drift by 15minute increments.

As you see this is very complicated stuff and since God is in the simplicity it is much better to just not use /etc/cron.* for whatever scripts and manually schedule each of the system cron jobs and custom scripts with cron at specific times.


4. Debian Linux time start schedule for cron.daily / cron.monthly / cron.weekly timing

As the last many years many of the servers I've managed were running Debian GNU / Linux, my first place to check was /etc/crontab which is the standard cronjobs file that is setting the { daily , monthly , weekly crons } 

 debian:~# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 фев 27 10:54 /etc/cron.d/
drwxr-xr-x 2 root root 4096 фев 27 10:55 /etc/cron.daily/
drwxr-xr-x 2 root root 4096 дек  7 23:04 /etc/cron.hourly/
drwxr-xr-x 2 root root 4096 дек  7 23:04 /etc/cron.monthly/
drwxr-xr-x 2 root root 4096 дек  7 23:04 /etc/cron.weekly/

debian:~# cat /etc/crontab 
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin# Example of job definition:
# .—————- minute (0 – 59)
# |  .————- hour (0 – 23)
# |  |  .———- day of month (1 – 31)
# |  |  |  .——- month (1 – 12) OR jan,feb,mar,apr …
# |  |  |  |  .—- day of week (0 – 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed
17 *    * * *    root    cd / && run-parts –report /etc/cron.hourly
25 6    * * *    root    test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.daily )
47 6    * * 7    root    test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.weekly )
52 6    1 * *    root    test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.monthly )

What above does is:

– Run cron.hourly once at every hour at 1:17 am
– Run cron.daily once at every day at 6:25 am.
– Run cron.weekly once at every day at 6:47 am.
– Run cron.monthly once at every day at 6:42 am.

As you can see if anacron is present on the system it is run via it otherwise it is run via run-parts binary command which is reading and executing one by one all scripts insude /etc/cron.hourly, /etc/cron.weekly , /etc/cron.mothly

anacron – few more words

Anacron is the canonical way to run at least the jobs from /etc/cron.{daily,weekly,monthly) after startup, even when their execution was missed because the system was not running at the given time. Anacron does not handle any cron jobs from /etc/cron.d, so any package that wants its /etc/cron.d cronjob being executed by anacron needs to take special measures.

If anacron is installed, regular processing of the /etc/cron.d{daily,weekly,monthly} is omitted by code in /etc/crontab but handled by anacron via /etc/anacrontab. Anacron's execution of these job lists has changed multiple times in the past:

debian:~# cat /etc/anacrontab 
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
HOME=/root
LOGNAME=root

# These replace cron's entries
1    5    cron.daily    run-parts –report /etc/cron.daily
7    10    cron.weekly    run-parts –report /etc/cron.weekly
@monthly    15    cron.monthly    run-parts –report /etc/cron.monthly

In wheezy and earlier, anacron is executed via init script on startup and via /etc/cron.d at 07:30. This causes the jobs to be run in order, if scheduled, beginning at 07:35. If the system is rebooted between midnight and 07:35, the jobs run after five minutes of uptime.
In stretch, anacron is executed via a systemd timer every hour, including the night hours. This causes the jobs to be run in order, if scheduled, beween midnight and 01:00, which is a significant change to the previous behavior.
In buster, anacron is executed via a systemd timer every hour with the exception of midnight to 07:00 where anacron is not invoked. This brings back a bit of the old timing, with the jobs to be run in order, if scheduled, beween 07:00 and 08:00. Since anacron is also invoked once at system startup, a reboot between midnight and 08:00 also causes the jobs to be scheduled after five minutes of uptime.
anacron also didn't have an upstream release in nearly two decades and is also currently orphaned in Debian.

As of 2019-07 (right after buster's release) it is planned to have cron and anacron replaced by cronie.

cronie – Cronie was forked by Red Hat from ISC Cron 4.1 in 2007, is the default cron implementation in Fedora and Red Hat Enterprise Linux at least since Version 6. cronie seems to have an acive upstream, but is currently missing some of the things that Debian has added to vixie cron over the years. With the finishing of cron's conversion to quilt (3.0), effort can begin to add the Debian extensions to Vixie cron to cronie.

Because cronie doesn't have all the Debian extensions yet, it is not yet suitable as a cron replacement, so it is not in Debian.
 

5. systemd-timers – The new crazy systemd stuff for script system job scheduling


Timers are systemd unit files with a suffix of .timer. systemd-timers was introduced with systemd so older Linux OS-es does not have it.
 Timers are like other unit configuration files and are loaded from the same paths but include a [Timer] section which defines when and how the timer activates. Timers are defined as one of two types:

 

  • Realtime timers (a.k.a. wallclock timers) activate on a calendar event, the same way that cronjobs do. The option OnCalendar= is used to define them.
  • Monotonic timers activate after a time span relative to a varying starting point. They stop if the computer is temporarily suspended or shut down. There are number of different monotonic timers but all have the form: OnTypeSec=. Common monotonic timers include OnBootSec and OnActiveSec.

    For each .timer file, a matching .service file exists (e.g. foo.timer and foo.service). The .timer file activates and controls the .service file. The .service does not require an [Install] section as it is the timer units that are enabled. If necessary, it is possible to control a differently-named unit using the Unit= option in the timer’s [Timer] section.

    systemd-timers is a complex stuff and I'll not get into much details but the idea was to give awareness of its existence for more info check its manual man systemd.timer

Its most basic use is to list all configured systemd.timers, below is from my home Debian laptop
 

debian:~# systemctl list-timers –all
NEXT                         LEFT         LAST                         PASSED       UNIT                         ACTIVATES
Tue 2020-03-24 23:33:58 EET  18s left     Tue 2020-03-24 23:31:28 EET  2min 11s ago laptop-mode.timer            lmt-poll.service
Tue 2020-03-24 23:39:00 EET  5min left    Tue 2020-03-24 23:09:01 EET  24min ago    phpsessionclean.timer        phpsessionclean.service
Wed 2020-03-25 00:00:00 EET  26min left   Tue 2020-03-24 00:00:01 EET  23h ago      logrotate.timer              logrotate.service
Wed 2020-03-25 00:00:00 EET  26min left   Tue 2020-03-24 00:00:01 EET  23h ago      man-db.timer                 man-db.service
Wed 2020-03-25 02:38:42 EET  3h 5min left Tue 2020-03-24 13:02:01 EET  10h ago      apt-daily.timer              apt-daily.service
Wed 2020-03-25 06:13:02 EET  6h left      Tue 2020-03-24 08:48:20 EET  14h ago      apt-daily-upgrade.timer      apt-daily-upgrade.service
Wed 2020-03-25 07:31:57 EET  7h left      Tue 2020-03-24 23:30:28 EET  3min 11s ago anacron.timer                anacron.service
Wed 2020-03-25 17:56:01 EET  18h left     Tue 2020-03-24 17:56:01 EET  5h 37min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service

8 timers listed.


N ! B! If a timer gets out of sync, it may help to delete its stamp-* file in /var/lib/systemd/timers (or ~/.local/share/systemd/ in case of user timers). These are zero length files which mark the last time each timer was run. If deleted, they will be reconstructed on the next start of their timer.

Summary

In this article, I've shortly explain logic behind debugging weird restart events etc. of Linux configured services such as Apache due to configured scripts set to run with a predefined scheduled job timing. I shortly explained on how to figure out why the preset default install configured cron jobs such as logrorate – the service that is doing system logs archiving and nulling run at a certain time. I shortly explained the mechanism behind cron.{daily, monthy, weekly} and its execution via anacron – runner program similar to crond that never misses to run a scheduled job even if a system downtime occurs due to a crashed Docker container etc. run-parts command's use was shortly explained. A short look at systemd.timers was made which is now essential part of almost every new Linux release and often used by system scripts for scheduling time based maintainance tasks.

Check if server is Physical Bare Metal or a Virtual Machine and its type

Tuesday, March 17th, 2020

check-if-linux-operating-system-is-running-on-physical-bare-metal-or-virtual-machine

In modern times the IT employee system administrator / system engineer / security engineer or a developer who has to develop and test code remotely on UNIX hosts, we have to login to multiple of different servers located in separate data centers around the world situated in Hybrid Operating system environments running multitude of different Linux OSes. Often especially for us sysadmins it is important to know whether the remote machine we have SSHed to is physical server (Bare Metal) or a virtual machines running on top of different kind of Hypervisor node OpenXen / Virtualbox / Virtuosso  / VMWare etc.
 

Then the question comes how to determine whether A remote Installed Linux is Physical or Virtual ?
 

1. Using the dmesg kernel log utility


The good old dmesg that is used to examine and control the kernel ring buffer detects plenty of useful information which gives you the info whether a server is Virtual or Bare Metal. It is present and accessible on every Linux server out there, thus using it is the best and simplest way to determine the OS system node type.

To grep whether a machine is Virtual and the Hypervisor type use:

nginx:~# dmesg | grep "Hypervisor detected"
[0.000000] Hypervisor detected: KVM


As you see above OS installed is using the KVM Virtualization technology.

An empty output of this command means the Remote OS is installed on a physical computer.

2. Detecting the OS platform the systemd way


Systemd along with the multiple over-complication of things that nearly all sysadmins (including me hate) so much introduced something useful in the fact of hostnamectl command
that could give you the info about the OS chassis platform.

 

root@pcfreak:~# hostnamectl status
 
 Static hostname: pcfreak
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 02425d67037b8e67cd98bd2800002671
           Boot ID: 34a83b9a79c346168082f7605c2f557c
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-amd64
      Architecture: x86-64

 

Below is output of a VM running on a Oracle Virtualbox HV.

linux:~# hostnamectl status
Static hostname: ubuntuserver
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 2befe86cf8887ca098f509e457554beb
 Boot ID: 8021c02d65dc46b1885afb25fddcf18c
 Virtualization: oracle
 Operating System: Ubuntu 16.04.1 LTS
 Kernel: Linux 4.4.0-78-generic
 Architecture: x86-64

3. Detect concrete container virtualization with systemd-detect-virt 


Another Bare Metal or VM identify tool that was introducted some time ago by freedesktop project is systemd-detect-virt (usually command is part of systemd package).
It is useful to detect the exact virtualization on a systemd running OS systemd-detect-virt is capable to detect many type of Virtualization type that are rare like: IBM zvm S390 Z/VM, bochs, bhyve (a FreeBSD hypervisor), Mac OS's parallels, lxc (linux containers), docker containers, podman etc.

The output from the command is either none (if no virtualization is present or the VM Hypervisor Host type):

server:~# systemd-detect-virt
none

quake:~# systemd-detect-virt
oracle

4. Install and use facter to report per node facts

debian:~# apt-cache show facter|grep -i desc -A2
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and
 reports per-node facts, which are collected by the Puppet agent and are made

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter
Tag: devel::lang:ruby, devel::library, implemented-in::ruby,
root@jeremiah:/home/hipo# apt-cache show facter|grep -i desc -A1
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter


– Install facter on Debian / Ubuntu / deb based Linux

# apt install facter –yes


– Install facter on RedHat / CentOS RPM based distros

# yum install epel-release

# yum install facter


– Install facter on OpenSuSE / SLES

# zypper install facter


Once installed on the system to find out whether the remote Operating System is Virtual:

# facter 2> /dev/null | grep virtual
is_virtual => false
virtual => physical


If the machine is a virtual machine you will get some different output like:

# facter 2> /dev/null | grep virtual
is_virtual => true
virtual => kvm


If you're lazy to grep you can use it with argument.

# facter virtual
physical

6. Use lshw and dmidecode (list hardware configuration tool)


If you don't have the permissions to install facter on the system and you can see whether lshw (list hardware command) is not already present on remote host.

# lshw -class system  
storage-host                  
    description: Computer
    width: 64 bits
    capabilities: smbios-2.7 vsyscall32

If the system is virtual you'll get an output similar to:

# lshw -class system  
debianserver 
 description: Computer
 product: VirtualBox
 vendor: innotek GmbH
 version: 1.2
 serial: 0
 width: 64 bits
 capabilities: smbios-2.5 dmi-2.5 vsyscall32
 configuration: family=Virtual Machine uuid=78B58916-4074-42E2-860F-7CAF39F5E6F5


Of course as it provides a verbosity of info on Memory / CPU type / Caches / Cores / Motherboard etc. virtualization used or not can be determined also with dmidecode / hwinfo and other tools that detect the system hardware this is described thoroughfully in my  previous article Get hardware system info on Linux.


7. Detect virtualziation using virt-what or imvirt scripts


imvirt is a little script to determine several virtualization it is pretty similar to virt-what the RedHat own script for platform identification. Even though virt-what is developed for RHEL it is available on other distros, Fedoda, Debian, Ubuntu, Arch Linux (AUR) just like is imvirt.

installing both of them is with the usual apt-get / yum or on Arch Linux with yay package manager (yay -S virt-what) …

Once run the output it produces for physical Dell / HPE / Fujitsu-Siemens Bare Metal servers would be just empty string.

# virt-what
#

Or if the system is Virtual Machine, you'll get the type, for example KVM (Kernel-based Virtual Machine) / virtualbox / qemu etc.

#imvirt
Physical

Conclusion


It was explained how to do a simple check whether the server works on a physical hardware or on a virtual Host hypervisor. The most basic and classic way is with dmesg. If no access to dmesg is due to restrictions you can try the other methods for systemd enabled OSes with hostnamectl / systemd-detect-virt. Other means if the tools are installed or you have the permissions to install them is with facter / lshw or with virt-what / imvirt scripts.
There definitely perhaps much more other useful tools to grasp hardware and virtualization information but this basics could be useful enough for shell scripting purposes.
If you know other tools, please share.
 

IBM TSM dsmc console client use for listing configured backups, checking set scheduled backups and backup and restore operations howto

Friday, March 6th, 2020

tsm-ibm-logo_tivoli-dsmc-console-client-listing-backups-create-backups-and-restore-on-linux-unix-windows

Creating a simple home based backup solution with some shell scripting and rsync is a common use. However as a sysadmin in a middle sized or large corporations most companies use some professional backup service such as IBM Tivoli Storage Manager TSM – recently IBM changed the name of the product to IBM Spectrum.

IBM TSM  is a data protection platform that gives enterprises a single point of control and administration for backup and recovery that is used for Privare Clouds backup and other high end solutions where data criticality is top.
Usually in large companies TSM backup handling is managed by a separate team or teams as managing a large TSM infrastructure is quite a complex task, however my experience as a sysadmin show me that even if you don't have too much of indepth into tsm it is very useful to know how to manage at least basic Incremental backup operations such as view what is set to be backupped, set-up a new directory structure for backup, check the backup schedule configured, check what files are included and which excluded from the backup store etc. 

TSM has multi OS support ans you can use it on most streamline Operating systems Windows / Mac OS X and Linux in this specific article I'll be talking concretely about backing up data with tsm on Linux, tivoli can be theoretically brought up even on FreeBSD machines via the Linuxemu BSD module and the 64-Bit Tivoli Storage Manager RPMs.
Therefore in this small article I'll try to give few useful operations for the novice admin that stumbles on tsm backupped server that needs some small maintenance.
 

1. Starting up the dsmc command line client

Nomatter the operating system on which you run it to run the client run:

# dsmc

tsm-check-backup-schedule-set-time

Note that usually dsmc should run as superuser so if you try to run it via a normal non-root user you will get an error message like:

[ user@linux ~]$ dsmc
ANS1398E Initialization functions cannot open one of the Tivoli Storage Manager logs or a related file: /var/tsm/dsmerror.log. errno = 13, Permission denied

Tivoli SM has an extensive help so to get the use basics, type help
 

tsm> help
1.0 New for IBM Tivoli Storage Manager Version 6.4
2.0 Using commands
  2.1 Start and end a client command session
    2.1.1 Process commands in batch mode
    2.1.2 Process commands in interactive mode
  2.2 Enter client command names, options, and parameters
    2.2.1 Command name
    2.2.2 Options
    2.2.3 Parameters
    2.2.4 File specification syntax
  2.3 Wildcard characters
  2.4 Client commands reference
  2.5 Archive
  2.6 Archive FastBack

Enter 'q' to exit help, 't' to display the table of contents,
press enter or 'd' to scroll down, 'u' to scroll up or
enter a help topic section number, message number, option name,
command name, or command and subcommand:    

2. Listing files listed for backups

A note to make here is as in most corporate products tsm supports command aliases so any command supported described in the help like query, could be
abbreviated with its first letters only, e.g. query filespace tsm cmd can be abbreviated as

tsm> q fi

Commands can be run non-interactive mode also so if you want the output of q fi you can straight use:

tsm> dsmc q fi

tsm-check-included-excluded-files-q-file-if-backupped-list-backup-set-directories

This shows the directories and files that are set for backup creation with Tivoli.

3. Getting included and excluded backup set files

It is useful to know what are the exact excluded files from tsm set backup this is done with query inclexcl

tsm-check-excluded-included-files

4. Querying for backup schedule time

Tivoli as every other backup solution is creating its set to backup files in a certain time slot periods. 
To find out what is the time slot for backup creation use;

tsm> q sched
Schedule Name: WEEKLY_ITSERV
      Description: ITSERV weekly incremental backup
   Schedule Style: Classic
           Action: Incremental
          Options: 
          Objects: 
         Priority: 5
   Next Execution: 180 Hours and 35 Minutes
         Duration: 15 Minutes
           Period: 1 Week  
      Day of Week: Wednesday
            Month:
     Day of Month:
    Week of Month:
           Expire: Never  

tsm-query-partitions-backupeed-or-not

5. Check which files have been backed up

If you want to make sure backups are really created it is a good to check, which files from the selected backup files have already
a working backup copy.

This is done with query backup like so:

tsm> q ba /home/*

tsm-dsmc-query-user-home-for-backups

If you want to query all the current files and directories backed up under a directory and all its subdirectories you need to add the -subdir=yes option as below:

tsm> q ba /home/hipo/projects/* -subdir=yes
   
Size      Backup Date        Mgmt Class A/I File
   —-      ———–        ———- — —-
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106
  1,024  08-12-2011 02:46:53    STANDARD    A  /home/hipo/projects/hsm41perf
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hsm41test
    512  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test/test2
 12,048  04-12-2011 02:01:29    STANDARD    A  /home/hipo/projects/hsm41perf/tables
 50,326  30-04-2012 01:35:26    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70023
 50,326  27-04-2012 00:28:15    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70099
 11,013  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg/md5check  

  • To make tsm, backup some directories on Linux / AIX other unices:

tsm> incr /  /usr  /usr/local  /home /lib

  • For tsm to backup some standard netware drives, use:

tsm> incr NDS:  USR:  SYS:  APPS:  

  • To backup C:\ D:\ E:\ F:\ if TSM is running on Windows

tsm> incr C:  D:  E: F:  -incrbydate 

  • To back up entire disk volumes irrespective of whether files have changed since the last backup, use the selective command with a wildcard and -subdir=yes as below:

tsm> sel /*  /usr/*   /home/*  -su=yes   ** Unix/Linux

7. Backup selected files from a backup location

It is intuitive to think you can just add some wildcard characters to select what you want
to backup from a selected location but this is not so, if you try something like below
you will get an err.

tsm> incr /home/hipo/projects/*/* -su=yes      
ANS1071E Invalid domain name entered: '/home/hipo/projects/*/*'


The proper way to select a certain folder / file for backup is with:

tsm> sel /home/hipo/projects/*/* -su=yes

8. Restoring tsm data from backup

To restore the config httpd.conf to custom directory use:

tsm> rest /etc/httpd/conf/httpd.conf  /home/hipo/restore/

N!B! that in order for above to work you need to have the '/' trailing slash at the end.

If you want to restore a file under a different name:

tsm> rest /etc/ntpd.conf  /home/hipo/restore/

9. Restoring a whole backupped partition

tsm> rest /home/*  /tmp/restore/ -su=yes

This is using the Tivoli 'Restoring multiple files and directories', and the files to restore '*'
are kept till the one that was recovered (saying this in case if you accidently cancel the restore)

10. Restoring files with back date 

By default the restore function will restore the latest available backupped file, if you need
to recover a specific file, you need the '-inactive' '-pick' options.
The 'pick' interface is interactive so once listed you can select the exact file from the date
you want to restore.

General restore command syntax is:
 

tsm> restore [source-file] [destination-file]


tsm> rest /home/hipo/projects/*  /tmp/restore/ -su=yes  -inactive -pick

TSM Scrollable PICK Window – Restore

     #    Backup Date/Time        File Size A/I  File
   ————————————————————————————————–
   170. | 12-09-2011 19:57:09        650  B  A   /home/hipo/projects/hsm41test/inclexcl.test
   171. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.ORIG
   172. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.TEST
   173. | 12-09-2011 19:57:09       1.13 KB  A   /home/hipo/projects/hsm41test/md5.out
   174. | 30-04-2012 01:35:26        512  B  A   /home/hipo/projects/hsm42125upg/PMR70023
   175. | 26-04-2012 01:02:08        512  B  I   /home/hipo/projects/hsm42125upg/PMR70023
   176. | 27-04-2012 00:28:15        512  B  A   /home/hipo/projects/hsm42125upg/PMR70099
   177. | 24-04-2012 19:17:34        512  B  I   /home/hipo/projects/hsm42125upg/PMR70099
   178. | 24-04-2012 00:22:56       1.35 KB  A   /home/hipo/projects/hsm42125upg/dsm.opt
   179. | 24-04-2012 00:22:56       4.17 KB  A   /home/hipo/projects/hsm42125upg/dsm.sys
   180. | 24-04-2012 00:22:56       1.13 KB  A   /home/hipo/projects/hsm42125upg/dsmmigfstab
   181. | 24-04-2012 00:22:56       7.30 KB  A   /home/hipo/projects/hsm42125upg/filesystems
   182. | 24-04-2012 00:22:56       1.25 KB  A   /home/hipo/projects/hsm42125upg/inclexcl
   183. | 24-04-2012 00:22:56        198  B  A   /home/hipo/projects/hsm42125upg/inclexcl.dce
   184. | 24-04-2012 00:22:56        291  B  A   /home/hipo/projects/hsm42125upg/inclexcl.ox_sys
   185. | 24-04-2012 00:22:56        650  B  A   /home/hipo/projects/hsm42125upg/inclexcl.test
   186. | 24-04-2012 00:22:56        670  B  A   /home/hipo/projects/hsm42125upg/inetd.conf
   187. | 24-04-2012 00:22:56       2.71 KB  A   /home/hipo/projects/hsm42125upg/inittab
   188. | 24-04-2012 00:22:56       1.00 KB  A   /home/hipo/projects/hsm42125upg/md5check
   189. | 24-04-2012 00:22:56      79.23 KB  A   /home/hipo/projects/hsm42125upg/mkreport.020423.out
   190. | 24-04-2012 00:22:56       4.27 KB  A   /home/hipo/projects/hsm42125upg/ssamap.020423.out
   191. | 26-04-2012 01:02:08      12.78 MB  A   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
   192. | 25-04-2012 16:33:36      12.78 MB  I   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
        0———10——–20——–30——–40——–50——–60——–70——–80——–90–
<U>=Up  <D>=Down  <T>=Top  <B>=Bottom  <R#>=Right  <L#>=Left
<G#>=Goto Line #  <#>=Toggle Entry  <+>=Select All  <->=Deselect All
<#:#+>=Select A Range <#:#->=Deselect A Range  <O>=Ok  <C>=Cancel
pick> 


To navigate in pick interface you can select individual files to restore via the number seen leftside.
To scroll up / down use 'U' and 'D' as described in the legenda.

 

11. Restoring your data to another machine

In certain circumstances, it may be necessary to restore some, or all, of your data onto a machine other than the original from which it was backed up.

In ideal case the machine platform should be identical to that of the original machine. Where this is not possible or practical please note that restores are only possible for partition types that the operating system supports. Thus a restore of an NTFS partition to a Windows 9x machine with just FAT support may succeed but the file permissions will be lost.
TSM does not work fine with cross-platform backup / restore, so better do not try cross-platform restores.
 Trying to restore files onto a Windows machine that have previously been backed up with a non-Windows one. TSM created backups on Windows sent by other OS platforms can cause  backups to become inaccessible from the host system.

To restore your data to another machine you will need the TSM software installed on the target machine. Entries in Tivoli configuration files dsm.sys and/or dsm.opt need to be edited if the node that you are restoring from does not reside on the same server. Please see our help page section on TSM configuration files for their locations for your operating system. 

To access files from another machine you should then start the TSM client as below:

# dsmc -virtualnodename=RESTORE.MACHINE      


You will then be prompted for the TSM password for this machine.

You will probably want to restore to a different destination to the original files to prevent overwriting files on the local machine, as below:

  • Restore of D:\ Drive to D:\Restore ** Windows 

tsm> rest D:\*   D:\RESTORE\    -su=yes 
 

  • Restore user /home/* to /scratch on ** Mac, Unix/Linux

tsm> rest /home/* /scratch/     -su=yes  
 

  • Restoring Tivoli data on old netware

tsm> rest SOURCE-SERVER\USR:*  USR:restore/   -su=yes  ** Netware

12. Adding more directories for incremental backup / Check whether TSM backup was done correctly?

The easiest way is to check the produced dschmed.log if everything is okay there should be records in the log that Tivoli backup was scheduled in a some hours time
succesfully.
A normally produced backup scheduled in log should look something like:

14-03-2020 23:03:04 — SCHEDULEREC STATUS BEGIN
14-03-2020 23:03:04 Total number of objects inspected:   91,497
14-03-2020 23:03:04 Total number of objects backed up:      113
14-03-2020 23:03:04 Total number of objects updated:          0
14-03-2020 23:03:04 Total number of objects rebound:          0
14-03-2020 23:03:04 Total number of objects deleted:          0
14-03-2020 23:03:04 Total number of objects expired:         53
14-03-2020 23:03:04 Total number of objects failed:           6
14-03-2020 23:03:04 Total number of bytes transferred:    19.38 MB
14-03-2020 23:03:04 Data transfer time:                    1.54 sec
14-03-2020 23:03:04 Network data transfer rate:        12,821.52 KB/sec
14-03-2020 23:03:04 Aggregate data transfer rate:        114.39 KB/sec
14-03-2020 23:03:04 Objects compressed by:                    0%
14-03-2020 23:03:04 Elapsed processing time:           00:02:53
14-03-2020 23:03:04 — SCHEDULEREC STATUS END
14-03-2020 23:03:04 — SCHEDULEREC OBJECT END WEEKLY_23_00 14-12-2010 23:00:00
14-03-2020 23:03:04 Scheduled event 'WEEKLY_23_00' completed successfully.
14-03-2020 23:03:04 Sending results for scheduled event 'WEEKLY_23_00'.
14-03-2020 23:03:04 Results sent to server for scheduled event 'WEEKLY_23_00'.

in case of errors you should check dsmerror.log
 

Conclusion


In this article I've briefly evaluated some basics of IBM Commercial Tivoli Storage Manager (TSM) to be able to  list backups, check backup schedules and how to the files set to be
excluded from a backup location and most importantly how to check that data backed up data is in a good shape and accessible.
It was explained how backups can be restored on a local and remote machine as well as how to  append new files to be set for backup on next incremental scheduled backup.
It was shown how the pick interactive cli interface could be used to restore files at a certain data back in time as well as how full partitions can be restored and how some
certain file could be retrieved from the TSM data copy.

How to enable HaProxy logging to a separate log /var/log/haproxy.log / prevent HAProxy duplicate messages to appear in /var/log/messages

Wednesday, February 19th, 2020

haproxy-logging-basics-how-to-log-to-separate-file-prevent-duplicate-messages-haproxy-haproxy-weblogo-squares
haproxy  logging can be managed in different form the most straight forward way is to directly use /dev/log either you can configure it to use some log management service as syslog or rsyslogd for that.

If you don't use rsyslog yet to install it: 

# apt install -y rsyslog

Then to activate logging via rsyslogd we can should add either to /etc/rsyslogd.conf or create a separte file and include it via /etc/rsyslogd.conf with following content:
 

Enable haproxy logging from rsyslogd


Log haproxy messages to separate log file you can use some of the usual syslog local0 to local7 locally used descriptors inside the conf (be aware that if you try to use some wrong value like local8, local9 as a logging facility you will get with empty haproxy.log, even though the permissions of /var/log/haproxy.log are readable and owned by haproxy user.

When logging to a local Syslog service, writing to a UNIX socket can be faster than targeting the TCP loopback address. Generally, on Linux systems, a UNIX socket listening for Syslog messages is available at /dev/log because this is where the syslog() function of the GNU C library is sending messages by default. To address UNIX socket in haproxy.cfg use:

log /dev/log local2 


If you want to log into separate log each of multiple running haproxy instances with different haproxy*.cfg add to /etc/rsyslog.conf lines like:

local2.* -/var/log/haproxylog2.log
local3.* -/var/log/haproxylog3.log


One important note to make here is since rsyslogd is used for haproxy logging you need to have enabled in rsyslogd imudp and have a UDP port listener on the machine.

E.g. somewhere in rsyslog.conf or via rsyslog include file from /etc/rsyslog.d/*.conf needs to have defined following lines:

$ModLoad imudp
$UDPServerRun 514


I prefer to use external /etc/rsyslog.d/20-haproxy.conf include file that is loaded and enabled rsyslogd via /etc/rsyslog.conf:

# vim /etc/rsyslog.d/20-haproxy.conf

$ModLoad imudp
$UDPServerRun 514​
local2.* -/var/log/haproxy2.log


It is also possible to produce different haproxy log output based on the severiy to differentiate between important and less important messages, to do so you'll need to rsyslog.conf something like:
 

# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log

Prevent Haproxy duplicate messages to appear in /var/log/messages

If you use local2 and some default rsyslog configuration then you will end up with the messages coming from haproxy towards local2 facility producing doubled simultaneous records to both your pre-defined /var/log/haproxy.log and /var/log/messages on Proxy servers that receive few thousands of simultanous connections per second.
This is a problem since doubling the log will produce too much data and on systems with smaller /var/ partition you will quickly run out of space + this haproxy requests logging to /var/log/messages makes the file quite unreadable for normal system events which are so important to track clearly what is happening on the server daily.

To prevent the haproxy duplicate messages you need to define somewhere in rsyslogd usually /etc/rsyslog.conf local2.none near line of facilities configured to log to file:

*.info;mail.none;authpriv.none;cron.none;local2.none     /var/log/messages

This configuration should work but is more rarely used as most people prefer to have haproxy log being written not directly to /dev/log which is used by other services such as syslogd / rsyslogd.

To use /dev/log to output logs from haproxy configuration in global section use config like:
 

global
        log /dev/log local2 debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

The log global directive basically says, use the log line that was set in the global section for whole config till end of file. Putting a log global directive into the defaults section is equivalent to putting it into all of the subsequent proxy sections.

Using global logging rules is the most common HAProxy setup, but you can put them directly into a frontend section instead. It can be useful to have a different logging configuration as a one-off. For example, you might want to point to a different target Syslog server, use a different logging facility, or capture different severity levels depending on the use case of the backend application. 

Insetad of using /dev/log interface that is on many distributions heavily used by systemd to store / manage and distribute logs,  many haproxy server sysadmins nowdays prefer to use rsyslogd as a default logging facility that will manage haproxy logs.
Admins prefer to use some kind of mediator service to manage log writting such as rsyslogd or syslog, the reason behind might vary but perhaps most important reason is  by using rsyslogd it is possible to write logs simultaneously locally on disk and also forward logs  to a remote Logging server  running rsyslogd service.

Logging is defined in /etc/haproxy/haproxy.cfg or the respective configuration through global section but could be also configured to do a separate logging based on each of the defined Frontend Backends or default section. 
A sample exceprt from this section looks something like:

#———————————————————————
# Global settings
#———————————————————————
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#———————————————————————
defaults
    mode                    tcp
    log                     global
    option                  tcplog
    #option                  dontlognull
    #option http-server-close
    #option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 7
    #timeout http-request    10s
    timeout queue           10m
    timeout connect         30s
    timeout client          20m
    timeout server          10m
    #timeout http-keep-alive 10s
    timeout check           30s
    maxconn                 3000

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

#———————————————————————
# frontend which proxys to the backends
#———————————————————————
frontend ft_DKV_PROD_WLPFO
    mode tcp
    bind 192.168.233.5:30000-31050
    option tcplog
    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
    default_backend Default_Bakend_Name


#———————————————————————
# round robin balancing between the various backends
#———————————————————————
backend bk_DKV_PROD_WLPFO
    mode tcp
    # (0) Load Balancing Method.
    balance source
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
    stick on src
    # (5) Server List
    # (5.1) Backend
    server Backend_Server1 10.10.10.1 check port 18088
    server Backend_Server2 10.10.10.2 check port 18088 backup


The log directive in above config instructs HAProxy to send logs to the Syslog server listening at 127.0.0.1:514. Messages are sent with facility local2, which is one of the standard, user-defined Syslog facilities. It’s also the facility that our rsyslog configuration is expecting. You can add more than one log statement to send output to multiple Syslog servers.

Once rsyslog and haproxy logging is configured as a minumum you need to restart rsyslog (assuming that haproxy config is already properly loaded):

# systemctl restart rsyslogd.service

To make sure rsyslog reloaded successfully:

systemctl status rsyslogd.service


Restarting HAproxy

If the rsyslogd logging to 127.0.0.1 port 514 was recently added a HAProxy restart should also be run, you can do it with:
 

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)


Or to restart use systemctl script (if haproxy is not used in a cluster with corosync / heartbeat).

# systemctl restart haproxy.service

You can control how much information is logged by adding a Syslog level by

    log         127.0.0.1 local2 info


The accepted values are the standard syslog security level severity:

Value Severity Keyword Deprecated keywords Description Condition
0 Emergency emerg panic System is unusable A panic condition.
1 Alert alert   Action must be taken immediately A condition that should be corrected immediately, such as a corrupted system database.
2 Critical crit   Critical conditions Hard device errors.
3 Error err error Error conditions  
4 Warning warning warn Warning conditions  
5 Notice notice   Normal but significant conditions Conditions that are not error conditions, but that may require special handling.
6 Informational info   Informational messages  
7 Debug debug   Debug-level messages Messages that contain information normally of use only when debugging a program.

Logging only errors / timeouts / retries and errors is done with option:

Note that if the rsyslog is configured to listen on different port for some weird reason you should not forget to set the proper listen port, e.g.:
 

  log         127.0.0.1:514 local2 info

option dontlog-normal

in defaults or frontend section.

You most likely want to enable this only during certain times, such as when performing benchmarking tests.

(or log-format-sd for structured-data syslog) directive in your defaults or frontend
 

Haproxy Logging shortly explained


The type of logging you’ll see is determined by the proxy mode that you set within HAProxy. HAProxy can operate either as a Layer 4 (TCP) proxy or as Layer 7 (HTTP) proxy. TCP mode is the default. In this mode, a full-duplex connection is established between clients and servers, and no layer 7 examination will be performed. When in TCP mode, which is set by adding mode tcp, you should also add option tcplog. With this option, the log format defaults to a structure that provides useful information like Layer 4 connection details, timers, byte count and so on.

Below is example of configured logging with some explanations:

Log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

haproxy-logged-fields-explained
Example of Log-Format configuration as shown above outputted of haproxy config:

Log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

haproxy_http_log_format-explained1

To understand meaning of this abbreviations you'll have to closely read  haproxy-log-format.txt. More in depth info is to be found in HTTP Log format documentation


haproxy_logging-explained

Logging HTTP request headers

HTTP request header can be logged via:
 

 http-request capture

frontend website
    bind :80
    http-request capture req.hdr(Host) len 10
    http-request capture req.hdr(User-Agent) len 100
    default_backend webservers


The log will show headers between curly braces and separated by pipe symbols. Here you can see the Host and User-Agent headers for a request:

192.168.150.1:57190 [20/Dec/2018:22:20:00.899] website~ webservers/server1 0/0/1/0/1 200 462 – – —- 1/1/0/0/0 0/0 {mywebsite.com|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/71.0.3578.80 } "GET / HTTP/1.1"

Haproxy Stats Monitoring Web interface


Haproxy is having a simplistic stats interface which if enabled produces some general useful information like in above screenshot, through which
you can get a very basic in browser statistics and track potential issues with the proxied traffic for all configured backends / frontends incoming outgoing
network packets configured nodes
 experienced downtimes etc.

haproxy-statistics-report-picture

The basic configuration to make the stats interface accessible would be like pointed in above config for example to enable network listener on address
 

https://192.168.0.5:8080/stats


with hproxyuser / password config would be:

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

Sessions states and disconnect errors on new application setup

Both TCP and HTTP logs include a termination state code that tells you the way in which the TCP or HTTP session ended. It’s a two-character code. The first character reports the first event that caused the session to terminate, while the second reports the TCP or HTTP session state when it was closed.

Here are some essential termination codes to track in for in the log:
 

Here are some termination code examples most commonly to see on TCP connection establishment errors:

Two-character code    Meaning
—    Normal termination on both sides.
cD    The client did not send nor acknowledge any data and eventually timeout client expired.
SC    The server explicitly refused the TCP connection.
PC    The proxy refused to establish a connection to the server because the process’ socket limit was reached while attempting to connect.


To get all non-properly exited codes the easiest way is to just grep for anything that is different from a termination code –, like that:

tail -f /var/log/haproxy.log | grep -v ' — '


This should output in real time every TCP connection that is exiting improperly.

There’s a wide variety of reasons a connection may have been closed. Detailed information about all possible termination codes can be found in the HAProxy documentation.
To get better understanding a very useful reading to haproxy Debug errors with  is in haproxy-logging.txt in that small file are collected all the cryptic error messages codes you might find in your logs when you're first time configuring the Haproxy frontend / backend and the backend application behind.

Another useful analyze tool which can be used to analyze Layer 7 HTTP traffic is halog for more on it just google around.

Procedure Instructions to safe upgrade CentOS / RHEL Linux 7 Core to latest release

Thursday, February 13th, 2020

safe-upgrade-CentOS-and_Redhat_Enterprise_Linux_RHEL-7-to-latest-stable-release

Generally upgrading both RHEL and CentOS can be done straight with yum tool just we're pretty aware and mostly anyone could do the update, but it is good idea to do some
steps in advance to make backup of any old basic files that might help us to debug what is wrong in case if the Operating System fails to boot after the routine Machine OS restart
after the upgrade that is usually a good idea to make sure that machine is still bootable after the upgrade.

This procedure can be shortened or maybe extended depending on the needs of the custom case but the general framework should be useful anyways to someone that's why
I decided to post this.

Before you go lets prepare a small status script which we'll use to report status of  sysctl installed and enabled services as well as the netstat connections state and
configured IP addresses and routing on the system.

The script show_running_services_netstat_ips_route.sh to be used during our different upgrade stages:
 

# script status ###
echo "STARTED: $(date '+%Y-%m-%d_%H-%M-%S'):" | tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
systemctl list-unit-files –type=service | grep enabled
systemctl | grep ".service" | grep "running"
netstat -tulpn
netstat -r
ip a s
/sbin/route -n
echo "ENDED $(date '+%Y-%m-%d_%H-%M-%S'):" | tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
####

– Save the script in any file like /root/status.sh

– Make the /root/logs directoriy.
 

[root@redhat: ~ ]# mkdir /root/logs
[root@redhat: ~ ]# vim /root/status.sh
[root@redhat: ~ ]# chmod +x /root/status.sh

1. Get a dump of CentOS installed version release and grub-mkconfig generated os_probe

[root@redhat: ~ ]# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
[root@redhat: ~ ]# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

2. Clear old versionlock marked RPM packages (if there are such)

On servers maintained by multitude of system administrators just like the case is inside a Global Corporations and generally in the corporate world , where people do access the systems via LDAP and more than a single person
has superuser privileges. It is a good prevention measure to use yum package management  functionality to RPM based Linux distributions called  versionlock.
versionlock for those who hear it for a first time is locking the versions of the installed RPM packages so if someone by mistake or on purpose decides to do something like :

[root@redhat: ~ ]# yum install packageversion

Having the versionlock set will prevent the updated package to be installed with a different branch package version.

Also it will prevent a playful unknowing person who just wants to upgrade the system without any deep knowledge to be able to
run

[root@redhat: ~ ]# yum upgrade

update and leave the system in unbootable state, that will be only revealed during the next system reboot.

If you haven't used versionlock before and you want to use it you can do it with:

[root@redhat: ~ ]# yum install yum-plugin-versionlock

To add all the packages for compiling C code and all the interdependend packages, you can do something like:

[root@redhat: ~ ]# yum versionlock gcc-*

If you want to clear up the versionlock, once it is in use run:

[root@redhat: ~ ]#  yum versionlock clear
[root@redhat: ~ ]#  yum versionlock list

3.  Check RPC enabled / disabled

This step is not necessery but it is a good idea to check whether it running on the system, because sometimes after upgrade rpcbind gets automatically started after package upgrade and reboot. 
If we find it running we'll need to stop and mask the service.

# check if rpc enabled
[root@redhat: ~ ]# systemctl list-unit-files|grep -i rpc
var-lib-nfs-rpc_pipefs.mount                                      static
auth-rpcgss-module.service                                        static
rpc-gssd.service                                                  static
rpc-rquotad.service                                               disabled
rpc-statd-notify.service                                          static
rpc-statd.service                                                 static
rpcbind.service                                                   disabled
rpcgssd.service                                                   static
rpcidmapd.service                                                 static
rpcbind.socket                                                    disabled
rpc_pipefs.target                                                 static
rpcbind.target                                                    static

[root@redhat: ~ ]# systemctl status rpcbind.service
● rpcbind.service – RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

[root@redhat: ~ ]# systemctl status rpcbind.socket
● rpcbind.socket – RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; disabled; vendor preset: enabled)
   Active: inactive (dead)
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)
           0.0.0.0:111 (Datagram)
           [::]:111 (Stream)
           [::]:111 (Datagram)

4. Check any previously existing downloaded / installed RPMs (check yum cache)

yum install package-name / yum upgrade keeps downloaded packages via its operations inside its cache directory structures in /var/cache/yum/*.
Hence it is good idea to check what were the previously installed packages and their count.

[root@redhat: ~ ]# cd /var/cache/yum/x86_64/;
[root@redhat: ~ ]# find . -iname '*.rpm'|wc -l

5. List RPM repositories set on the server

 [root@redhat: ~ ]# yum repolist
Loaded plugins: fastestmirror, versionlock
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Determining fastest mirrors
repo id                                                                                 repo name                                                                                                            status
!atos-ac/7/x86_64                                                                       Atos Repository                                                                                                       3,128
!base/7/x86_64                                                                          CentOS-7 – Base                                                                                                      10,019
!cr/7/x86_64                                                                            CentOS-7 – CR                                                                                                         2,686
!epel/x86_64                                                                            Extra Packages for Enterprise Linux 7 – x86_64                                                                          165
!extras/7/x86_64                                                                        CentOS-7 – Extras                                                                                                       435
!updates/7/x86_64                                                                       CentOS-7 – Updates                                                                                                    2,500

This step is mandatory to make sure you're upgrading to latest packages from the right repositories for more concretics check what is inside in confs /etc/yum.repos.d/ ,  /etc/yum.conf 
 

6. Clean up any old rpm yum cache packages

This step is again mandatory but a good to follow just to have some more clearness on what packages is our upgrade downloading (not to mix up the old upgrades / installs with our newest one).
For documentation purposes all deleted packages list if such is to be kept under /root/logs/yumclean-install*.out file

[root@redhat: ~ ]# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

7. List the upgradeable packages's latest repository provided versions

[root@redhat: ~ ]# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Then to be aware how many packages we'll be updating:

[root@redhat: ~ ]#  yum check-update | wc -l

8. Apply the actual uplisted RPM packages to be upgraded

[root@redhat: ~ ]# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Again output is logged to /root/logs/yumcheckupate-*.out 

9. Monitor downloaded packages count real time

To make sure yum upgrade is not in some hanging state and just get some general idea in which state of the upgrade is it e.g. Download / Pre-Update / Install  / Upgrade/ Post-Update etc.
in mean time when yum upgrade is running to monitor,  how many packages has the yum upgrade downloaded from remote RPM set repositories:

[root@redhat: ~ ]#  watch "ls -al /var/cache/yum/x86_64/7Server/…OS-repository…/packages/|wc -l"

10. Run status script to get the status again

[root@redhat: ~ ]# sh /root/status.sh |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

11. Add back versionlock for all RPM packs

Set all RPM packages installed on the RHEL / CentOS versionlock for all packages.

#==if needed
# yum versionlock \*

12. Get whether old software configuration is not messed up during the Package upgrade (Lookup the logs for .rpmsave and .rpmnew)

During the upgrade old RPM configuration is probably changed and yum did automatically save .rpmsave / .rpmnew saves of it thus it is a good idea to grep the prepared logs for any matches of this 2 strings :
 

[root@redhat: ~ ]#   grep -i ".rpm" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out
[root@redhat: ~ ]#  grep -i ".rpmsave" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out
[root@redhat: ~ ]#  grep -i ".rpmnew" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out


If above commands returns output usually it is fine if there is is .rpmnew output but, if you get grep output of .rpmsave it is a good idea to review the files compare with the original files that were .rpmsaved with the 
substituted config file and atune the differences with the changes manually made for some program functionality.

What are the .rpmsave / .rpmnew files ?
This files are coded files that got triggered by the RPM install / upgrade due to prewritten procedures on time of RPM build.

If a file was installed as part of a rpm, it is a config file (i.e. marked with the %config tag), you've edited the file afterwards and you now update the rpm then the new config file (from the newer rpm) will replace your old config file (i.e. become the active file).
The latter will be renamed with the .rpmsave suffix.

If a file was installed as part of a rpm, it is a noreplace-config file (i.e. marked with the %config(noreplace) tag), you've edited the file afterwards and you now update the rpm then your old config file will stay in place (i.e. stay active) and the new config file (from the newer rpm) will be copied to disk with the .rpmnew suffix.
See e.g. this table for all the details. 

In both cases you or some program has edited the config file(s) and that's why you see the .rpmsave / .rpmnew files after the upgrade because rpm will upgrade config files silently and without backup files if the local file is untouched.

After a system upgrade it is a good idea to scan your filesystem for these files and make sure that correct config files are active and maybe merge the new contents from the .rpmnew files into the production files. You can remove the .rpmsave and .rpmnew files when you're done.


If you need to get a list of all .rpmnew .rpmsave files on the server do:

[root@redhat: ~ ]#  find / -print | egrep "rpmnew$|rpmsave$

13. Reboot the system 

To check whether on next hang up or power outage the system will boot normally after the upgrade, reboot to test it.

you can :

[root@redhat: ~ ]#  reboot

either

[root@redhat: ~ ]#  shutdown -r now


or if on newer Linux with systemd in ues below systemctl reboot.target.

[root@redhat: ~ ]#  systemctl start reboot.target

14. Get again the system status with our status script after reboot

[root@redhat: ~ ]#  sh /root/status.sh |tee /root/logs/status-after-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

15. Clean up any versionlocks if earlier set

[root@redhat: ~ ]# yum versionlock clear
[root@redhat: ~ ]# yum versionlock list

16. Check services and logs for problems

After the reboot Check closely all running services on system make sure every process / listening ports and services on the system are running fine, just like before the upgrade.
If the sytem had firewall,  check whether firewall rules are not broken, e.g. some NAT is not missing or anything earlier configured to automatically start via /etc/rc.local or some other
custom scripts were run and have done what was expected. 
Go through all the logs in /var/log that are most essential /var/log/boot.log , /var/log/messages … yum.log etc. that could reveal any issues after the boot. In case if running some application server or mail server check /var/log/mail.log or whenever it is configured to log.
If the system runs apache closely check the logs /var/log/httpd/error.log or php_errors.log for any strange errors that occured due to some issues caused by the newer installed packages.
Usually most of the cases all this should be flawless but a multiple check over your work is a stake for good results.
 

How to install jcmd on CentOS 7 to diagnose running Java Virtual Machine crashing applications

Tuesday, January 14th, 2020

how-to-install-jcmd-centos-jcdm-java-logo-1

jcmd utility is well known in the Brane New wonderful world of Java but if you're like me a classical old school sysadmins and non-java developer you probably never heard it hence before going straight into how to install it on CentOS 7 Linux servers, I'll shortly say few words on what it is.
jcmd is used to send diagnostic requests to running Java Virtual Machine (JVM) it is available in both in Oracle Java as well as OpenJDK.
The requests jcmd sends to VM are based on the running Java PID ID and are pretty useful for controlling Java Flight Recordings, troubleshoot, and diagnose JVM and Java Applications. It must be used on the same machine where the JVM is running.

 Used without arguments or with the -l option, jcmd prints the list of running Java processes with their process id, their main class and their command line arguments.
When a main class is specified on the command line, jcmd sends the diagnostic command request to all Java processes for which the command line argument is a substring of the Java process' main class.
jcmd could be useful if the JConsoleJMX (Java Management Extensions) can't be used for some reason on the server or together with Java Visual VM (visual interface for viewing detailed info about Java App).

In most Linux distributionsas as  of year 2020 jcmd is found in  java-*-openjdk-headless.

To have jcmd on lets say Debian GNU / Linux, you're up to something like:

apt-get install –yes openjdk-12-jdk-headless
 

or

apt-get install openjdk-11-jdk-headless
however in CentOS
7 jcmd is not found in java*openjdk*headless but instead to have it on server, thus it take me a while to look up where it is foundso after hearing
from some online post it is part of package java*openjdk*devel* to make sure this so true, I've used the  –download-only option
 

 yum install –downloadonly –downloaddir=/tmp java-1.8.0-openjdk-devel


So the next question was how to inspect the downloaded rpm package into /tmp usually, this is possible via Midnight Commander (mc) easily to view contents, however as this
server did not have installed mc due to security policies I had to do it differently after pondering a while on how to to list the RPM package file content come up using following command

 rpm2cpio java-1.8.0-openjdk-devel-1.8.0.232.b09-0.el7_7.x86_64.rpm |cpio -idmv|grep -i jcmd|less


To then install  java-1.8.0-openjdk-devel-1.8.0.232.b09-0.el7_7.x86_64.rpm to do so run:

yum -y install  java-1.8.0-openjdk-devel-1.8.0*


Once the jcmd, I've created the following bash script  that was set that was tracking for application errors and checking whether the JBoss application server pool-available-count is not filled up and hence jboss refuses to serve connections  through jboss-cli.sh query automatically launching jcmd to get various diagnostic data about Java Virtual Machine (e.g. a running snapshot) – think of it like the UNIX top for debugging or Windows System Monitor but run one time. 

# PID_OF_JAVA=$(pgrep -l java)
# jcmd $PID_OF_JAVA GC.heap_dump GC.heap_dump_file-$(date '+%Y-%m-%d_%H-%M-%S').jfr
# jcmd $PID_OF_JAVA Thread.print > Thread.print-$(date '+%Y-%m-%d_%H-%M-%S').jfr


The produced log files can then be used by the developer to visualize some Java specific stuff "Flight recordings" like in below screenshot:

jcmd-java-tool-dump-jfr-flight-recordings-visualized-java 

If you're interested on some other interesting tools that can be used to Monitor  and Debug a Running Java VM take a look at Java's official documentation Monitoring Tools.
So that's all Mission Accomplished 🙂 Now the Java Application developer could observe the log and tell why exactly the application crashed after the multitude of thrown Exceptions in the JBoss server.log.