The Timekeeper of Ukraine
11 min read
For six years, Vladimir Soldatov has been the custodian of Ukraine’s time. He oversees a laboratory in the city of Kharkiv that contains about a dozen clocks and several distributive devices: gray boxes, humming in gray racks and connected via looping cables, that together create, count, and communicate his country’s seconds. The lab is located within the Institute of Metrology, a cluster of cream-colored buildings now scarred by Russian artillery.
Soldatov is Ukraine’s representative in a small, international community of obsessives who keep their nation’s time and, by doing so, help construct the world’s time, to which all clocks are set. The timekeepers compare their labs’ outputs once every five days; many then tweak their systems in increments of trillionths of a second. In the digital era, no such lab has operated in a war zone until now.
Kharkiv has endured waves of bombardment since Russia invaded Ukraine on February 24, 2022. During that first winter, an explosion about 50 meters from Soldatov’s lab shattered all of its windows and spiked its herringbone wooden floors with shards of glass. Most of the lab’s devices kept ticking. The windows are now filled with wood and insulation, and Soldatov and his colleagues have moved many of their instruments from second- and third-floor rooms to a basement space, Soldatov explained to me by email, “in case the building is directly hit again.”
Modern timekeeping is a science of nearly unfathomable precision, built on counting the perfectly steady, rapidly recurring undulations of energy waves. An energy wave rises, peaks, descends, bottoms out, ascends, and returns to its original altitude: tick. The wave repeats the same motion, at the same pace, taking the same interval of time to complete the cycle: tick. The trick for timekeepers is to ensure that they all use waves that oscillate at the same rate—the same frequency.
Since 1967, when timekeepers officially forsook astronomy for physics as the basis of the second, the time’s definitional energy wave has been one that, when it hits a cesium-133 atom that is in one of two energy states, inspires the atom to switch to the other. This change is called a “clock transition.” To find that frequency, a cesium clock embarks on a continuous search. It uses magnets or lasers to select only the atoms that are in one of those two states, beams them through a pair of energy fields, and notes how many atoms make the leap. The clock then adjusts the frequency of the fields’ energy and sends another batch of atoms through—repeating this process over and over, nearing, then overshooting, then nearing, then overshooting the frequency. The clock determines the frequency of energy that seems to convert the most atoms, then counts its undulations.
For these measurements to be as accurate as possible, timekeepers have to sustain a stable environment around the atoms, both in the lab and within the clocks, which are also known as “standards.” Some timekeeping labs keep their temperature constant within tenths of a degree Celsius. The clocks have their own internal temperature stabilizers, as well as vacuums that remove excess atoms and molecules, and shields to fend off energy waves that could distort their readings. The need to control for every variable can hardly be overstated, because the second is arguably civilization’s most precisely defined unit of measurement. Ideally, each second should contain 9,192,631,770 ticks of the definitional frequency.
Not all atomic clocks use cesium. In the Kharkiv lab, three clocks do; six use hydrogen, which can be more stable in the short term, and several use rubidium, though none of the latter contribute to Ukraine’s time. Today, that is the job of two cesium and two hydrogen clocks. Another hydrogen clock operates in reserve. Soldatov previously kept four reserves running, but that would require more electricity than he can currently use. Of his work, Soldatov told me, “Mostly, I have to repair the old Soviet hydrogen standards.” Sometimes, he has to disassemble devices to adjust their inner sanctum, often using spare parts. And, he added, “the cesium standard doesn’t like it when a rocket explodes 50 meters away.”
That early blast tore off the side walls of several hydrogen clocks, badly bending them, but they kept running. The cesium clock closest to the wall, though, stopped transmitting its time to the measuring system. Soldatov shut it down. When he started it back up, it lagged behind the other clocks. He restarted it again, under the manufacturer’s advice, but then it rushed. He tried again. For a while, it seemed to operate normally, but then it began performing too poorly to keep running.
Under ordinary circumstances, all clocks accelerate or decelerate at their own rate. Even atomic ones made to the exact same specifications will deviate from one another. Partly for this reason, timekeeping labs generally use a weighted average of their clocks’ readings, called a “timescale,” as their time. (Another reason is that any clock can fail.)
The world’s central timescale, called Coordinated Universal Time (UTC), is based on the weighted average of more than 400 clocks in about 70 labs across the world. A seven-person department within the International Bureau of Weights and Measures, in the Paris suburb of Sèvres, calculates this average. (In the latest version, clocks from Russia’s metrology lab contributed the highest percentage of the time.) One country’s time is seldom in sync with another’s; even when they do coincide, they don’t remain that way for long. But the best timekeeping labs tend to steer their time to keep within a couple nanoseconds of UTC; others stay within hundreds or thousands. Soldatov has generally kept Ukraine’s time within about 20 nanoseconds.
At the beginning of the war, around the time the blast shattered the institute’s windows—causing the lab’s temperature to sharply drop—Ukraine’s time rose to more than 65 billionths of a second fast and stayed there for 10 days before recovering. “I had no time to monitor the scale,” Soldatov said. In another period, when the city went without power for several days, the hydrogen standards had to be shut off to conserve energy, and so Ukraine’s time jumped again to nearly 40 nanoseconds ahead. The lab’s latest differences, for the five-day intervals between July 28 and August 27, have ranged from 3.8 billionths of a second fast to 2.1 slow, though for several stretches Soldatov was unable to submit his data in time.
Soldatov is 46 years old, grew up in Kharkiv, and graduated from its Air Force University with a degree in radio engineering. “I became the custodian somewhat by chance,” he wrote to me. After serving in the army for 10 years, he worked as a programmer and built websites. “A friend invited me to set up computers and write a program for a frequency synthesizer for the system, and after some time, I became the head of the laboratory.”
When the Russians began bombing Kharkiv, Soldatov directed most of the laboratory’s staff to stay home. One colleague remained with him to keep the time: Demian Mykolayovych Kravchenko, an engineer who moved with his family into the institute’s subterranean bomb shelter within days of the Russian incursion to escape the relentless shelling of their neighborhood.
A time relies on having someone to look after it. If a lab’s clocks do not operate continuously, the time will be lost. Many timekeepers swear that the devices wait until nights or weekends to malfunction. “It may sound funny, but I treat the system as a living organism,” Soldatov joked. He thinks of his instruments as temperamental colleagues, some of whose components are not much younger than he is. They harbor a furtive mischief: “If a staff member leaves the laboratory, something breaks. I can’t explain it, but it happens.”
In the early days of the war, the Russians bombed the city mainly at night. Soldatov often stayed overnight at the institute to tend to the standards, and especially to keep an eye on the generator if the power went out. Kravchenko sometimes helped with the whole building’s engineering needs; the institute’s then-director later described him as “a true guardian angel of the Institute of Metrology.”
But the Russians changed their patterns, according to Soldatov, first to random times within the city’s curfew, but then to any time, including during the day and in crowded places, “due to the great efficiency and desire of the Russian Federation to kill as many people as possible.” One Sunday morning, Kravchenko was in the institute’s back area when a cluster bomb hit. “He was killed in front of his family,” Soldatov told me. Soldatov had spoken with Kravchenko at the institute just hours before.
For many months afterward, Soldatov kept Ukraine’s time alone. Now he has a staff again, and he works remotely except when the devices need to be reset or repaired, or when he comes in to develop devices for the future.
In recent weeks, Russia has redoubled its attacks on Kharkiv, raising a question that I put to Soldatov: What would happen if Ukraine’s primary timekeeping lab were destroyed? What if a national time suddenly disappeared?
His first response was sobering. “We have not conducted experiments to determine what would happen if the single point of synchronization fails,” he wrote, “but it is highly likely that achieving high-speed internet and stable communication will be impossible. Additionally, there will be issues with electricity and frequency stability in the network.” But then he amended that, writing later: “I don’t think there will be any catastrophic consequences from the destruction of the clocks.”
Timekeeping may be delicate, but it is also resilient, because its burden is distributed and shared. Critical infrastructure all over the world relies on numerous clocks in far-flung places to remain synchronized within millionths of a second. Power grids, for instance, use temporal alignment to pinpoint failures. But a grid’s clocks don’t need to be synchronized to UTC or even a national lab. They simply need to be synchronized to one another. Ukrenergo, the Ukrainian grid’s operator, synchronizes its substations using readings not from the lab in Kharkiv but beamed down from GPS.
Many telecommunications providers around the world operate similarly: Their networks need to stay synchronized to connect calls across towers, and they often do so using GPS receivers and clocks. Exactly how, and to what time, Ukrainian telecoms companies synchronize is so essential to their functioning that three of the country’s providers declined to describe their timekeeping systems to me. A representative of Lifecell responded to my query, “The information you are requesting is quite sensitive and cannot be disclosed, especially during the war.”
The Institute of Metrology is linked directly to the internet. Soldatov’s lab houses two servers that distribute Ukrainian time to anyone who wants it. Together, they receive about a quarter million requests for the time a day, “sometimes many more,” Soldatov said. Because these servers are connected to a primary time source, they are at the top of the internet’s temporal hierarchy, on stratum one. Beneath them, on stratum two, are go-between servers that pass their understanding of the time along to other servers and machines. And so the time trickles through the web, often synchronous within tens of milliseconds, down to the innumerable devices that sustain the internet and, degrading as it goes, to the corners of the screens of the public’s personal computers.
This system was designed in the earliest days of the internet, when network devices failed frequently. It’s based on principles that are fundamental to timekeeping: redundancy and diversity. If the Institute of Metrology’s servers cut out, any server looking to them should be programmed to also seek the time from at least two other sources. Whether system administrators have properly set up these processes would be revealed only if the worst happened.
Coders have often been surprised by how complex the time can be. Once in a while, a leap second must be added to UTC to keep the time mostly aligned with the Earth’s erratic rotation. When this happens, websites and digital systems have been known to fail, because of a gap of a single second. Without sources of time like the lab in Ukraine, improperly programmed systems would swiftly drift at least a second out of sync. Encrypted systems would especially suffer, as they require a particularly large amount of synchronous data to operate. Websites could break, or at least slow.
In Ukraine, rolling blackouts already limit the hours that people can spend online communicating, working, or reading. Losing a primary, central time source could cause an additional disturbance to internet access across the nation. The country’s handful of surviving stratum-one servers could be flooded with requests. Whether they would be overwhelmed or hold steady is not known.
Ukraine does have a lesser, backup timescale in Kyiv, and the Ukrainian military has its own standards. But if the Kharkiv lab were destroyed, Ukraine would almost certainly depend more than ever on GPS for the time. The country’s stratum-one servers outside the lab use it as their time source, just as the power grid does. In a 2019 paper, Soldatov warned against Ukraine’s reliance on GPS for the time. “According to some experts, our dependence on GPS is becoming very dangerous, given the extreme unreliability of this technology,” he wrote. “The problem is that the signal from the satellite is very weak, and it is extremely easy to muffle it with generating noise at the same frequency.” The war has made GPS even harder to access in Ukraine.
Soldatov does not believe that the war will destroy his lab’s timescale. But he has suffered other losses since it began. A rocket badly damaged his family’s home, and so he, his wife, and his teenage son now live with his mother in a small apartment, a few kilometers from the lab.
“Recently, one of the hydrogen standards just went out,” Soldatov wrote to me. “That same day, the daughter of a colleague called me and said that her father had died.” This colleague had worked specifically on that device. He had been struggling with his health, but Soldatov believes the war accelerated his decline. “He died around the same time the hydrogen standard went out,” Soldatov told me. He has since repaired the standard.
Soldatov once saw his work as being central to Ukraine’s technological ascendence and an indicator of “technical potential.” He wrote, “The better the clock in the country, the more developed it is, as a rule, the more data it can process.” But he has come to believe that Ukraine’s potential is withering. “Yes, sometimes it seems to me that all my work is meaningless and has few prospects, and I want to go somewhere far away and do my own thing, but for now I am where I am.” During an earlier stretch of the war, Soldatov sent his wife and son elsewhere, but he stayed.
Soldatov learned in the military to value perseverance and responsibility over self-preservation, he told me, and he thinks that most Kharkiv residents share this mentality. “If the clocks are destroyed,” he wrote, “I will go to serve on the front lines.”
