Ulysse

Cannot sync with NTP server

Recommended Posts

Hi,

I am having troubles connecting my Sawyer to a local NTP server.

I have a local NTP server on my network that I know to be working : I can query it using some Linux machine and get offset, jitter, etc.
I want to connect Sawyer to it so that it is in sync with my development computer (otherwise I can't use MoveIt! nor tf, pretty embarrassing...)

I followed the Time and NTP tutorial, booted Sawyer into the FSM, replaced the NTP pool list with the IP address of my NTP server and rebooted into the SDK.
However it didn't work, timestamps published by the robot are still off by about 6 hours.

For completeness:

  • The robot Intera version 5.2.2
  • The development computer Intera version is 5.2
  • I work in France so I am on GMT+2 timezone
  • The `ntpdate` command does not work, it raises the error:
server 192.168.212.60, stratum 0, offset 0.000000, delay 0.00000
11 Jun 10:28:25 ntpdate[4829]: no server suitable for synchronization found
  • The `clockdiff` command works and outputs the above offset:
host=192.168.212.60 rtt=562(280)ms/0ms delta=-17953355ms/-17953355ms Mon Jun 11 10:54:02 2018
  • I spent some time on the FSM trying to get this working and I might have disrupted some parameters. For instance, the `ntpdate` was working at first but it no longer does.

How can I investigate on why is the robot not connecting to the server ?
Normally it should burst on it at boot time as the offset is way bigger than 17 minutes.

Thanks for your help.

Share this post


Link to post
Share on other sites

Hi Ulysse,

I'd recommend using some of example settings in the tutorial for your workstation's computer:

# editing your workstation's /etc/ntp.conf
# remove or comment out your workstation's references to other NTP servers or NTP pools

# add backup fake local server
server 127.127.1.0 iburst      # use local clock of sdk workstation as the server, burst synchronize
fudge 127.127.1.0  stratum 10  # "fake" sdk workstation as a stratum 10 NTP server (rather than the default and invalid stratum 16)

# set permissions to use your computer as an NTP server
restrict -4 default kod notrap nomodify nopeer limited  # remove the default "noquery" options to allow others on the LAN to see the NTP server
restrict -6 default kod notrap nomodify nopeer limited  # ditto

# optionally broadcast your IP address to the network
broadcast <your>.<ip>.<address>.<here>

Of particular interest, you'll need to "fudge" a realistic stratum like 10 to convince other NTP clients to use your clock, unless you actually have a stratum 0 atomic clock. I hope this help!

~ Ian

Share this post


Link to post
Share on other sites

Please, I've been pulling my hair out on this for two weeks, obviously I tried everything that's on the tutorial.
I'm sorry for being rude, I totally understand that you need to post the obvious solutions for other users that might come here.
But in the meantime, can you also provide me with a real support ? My issue is not trivial and has been blocking my work for weeks...

As a sign of goodwill I tried again the solution you proposed : it still isn't working, the offset keeps oscillating around (minus) 5 hours.

Share this post


Link to post
Share on other sites

Hi Ulysse,

A large portion of the time, these tough technical problems are solved by a simple change somewhere along the way, which is why I started from the beginning with my recommendations. I'm sorry to hear that this issue has been frustrating you, and we will do our best to help.

First, a few questions to help me understand your network topology: Are you on a university network? Are your machines connected to the Internet? Which machine is acting as the NTP server? How are you networking your workstation to the Sawyer?

You've mentioned that the robot is off by about 5-6 hours in time. If I'm not mistaken, you're located in France in GMT+2. The robot's default timezone is EST, which is currently GMT-4. Is it possible that the timezone of your robot is still set to EST? That would put your robot exactly 6 hours behind. You can check and modify this in the FSM:
http://sdk.rethinkrobotics.com/intera/Field_Service_Menu_(FSM)#Menu_Functions

Best,
~ Ian

Share this post


Link to post
Share on other sites

It is an enterprise network on a VLAN, using a separate IP address range: 192.168.212.0/22.
There is no internet access as no gateway has been configured (FSM won't let me leave the field empty so I have to type a dummy one).
Lastly there is a NTP server on the network whose IP address is 192.168.212.22. I know it is working as my workstation computer is in sync with it.

I double-checked the timezone in the FSM and it IS set to Europe/Paris. But I don't think this really matters as ROS timestamps are in nanoseconds since epoch, which is timezone independent. Furthermore the offset isn't exactly 5 hours (I know I wrote 6 earlier on, my bad), it is 5 hours minus ~40 seconds.

Thanks for your help.

Share this post


Link to post
Share on other sites

You mention in your last comment that the NTP server is at 192.168.212.22. In the initial post your workstation configuration file states the server is at 192.168.212.60.

Which IP do you have in Sawyer's FSM under NTP Servers? I have a screencrap of the field in question attached.

Just so I have my information correct, could you list out the local IP addresses of the robot, user workstation, and NTP servers? And could you post the contents of your workstation's NTP configuration file?

Thank you,
~ Ian

just-ntp-paris.png

Share this post


Link to post
Share on other sites

Sorry I was confusing, my examples using `ntpdate` and `clockdiff` were to check clock offsets between my workstation and the Sawyer.
For clarity (all IP addresses are configured statically):

  • NTP server: 192.168.212.22
  • Sawyer: 192.168.212.60
  • Workstation (Ubuntu 16.04 LTS virtual machine running under Oracle VirtualBox on Windows 7 Enterprise): 192.168.212.160

Content of my workstation's `/etc/ntp.conf` file:

# ntp.conf: Managed by puppet.
#
# Enable next tinker options:
# panic - keep ntpd from panicking in the event of a large clock skew
# when a VM guest is suspended and resumed;
# stepout - allow ntpd change offset faster
tinker panic 0

disable monitor

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict -4 default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict ::1

# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# iburst - allow send sync packages faster if upstream unavailable
# prefer - select preferrable server
# minpoll - set minimal update frequency
# maxpoll - set maximal update frequency
server 192.168.212.22 iburst prefer
server 10.8.113.254 iburst
server time.my.company.fr iburst

# Driftfile.
driftfile /var/lib/ntp/drift

Some outputs from my workstation:

user@workstation:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.212.22  132.166.192.1    3 u   32   64    0    0.000    0.000   0.000
+10.8.113.254    192.168.8.2      3 u    4   64    7    1.117    4.700  10.553
*my.company      195.220.94.163   2 u    3   64    7    1.154   -2.642  11.157
user@workstation:~$ ntpdate -q 192.168.212.22
server 192.168.212.22, stratum 3, offset -0.062119, delay 0.02606
13 Jun 09:37:19 ntpdate[2368]: no server suitable for synchronization found
user@workstation:~$ ntpdate -q 192.168.212.60
server 192.168.212.60, stratum 0, offset 0.000000, delay 0.00000
13 Jun 09:38:24 ntpdate[2373]: no server suitable for synchronization found
user@workstation:~$ clockdiff -o 192.168.212.60
......
host=192.168.212.60 rtt=177(265)ms/0ms delta=-17952260ms/-17952260ms Wed Jun 13 09:38:58 2018

Content of Sawyer's FSM:

  • Hostname: 02170CP00029
  • IP type: Static IP
  • IP Address*: 192.168.212.60
  • IP Netmask*: 255.255.252.0
  • IP Gateway*: 192.168.215.254 # That is a dummy one as there is no gateway configured on my network
  • DNS: # Empty
  • NTP Servers: 192.168.212.22
  • ROS Naming Type: ROS_IP
  • Timezone: Europe/Paris

Let me know if you need further information. Thanks Ian.

Share this post


Link to post
Share on other sites

That is very helpful information. I think I now have a good handle on your network topology. Thank you Ulysse. After looking through your information, I realized it would be helpful to see your NTP Server's config file. Is there any chance you could supply that? Also, is this NTP server connected to the Internet or an absolute time device?

Thank you,
~ Ian

Share this post


Link to post
Share on other sites

I cannot have access to the NTP server but here is what I know about it:

  • OS is CentOS Linux release 7.2
  • NTP version is 4.2.6p5
  • It uses the `time.my.company.fr` server as a reference, which is connected to the Internet

Do you have specific informations you want me to ask for?

Thanks Ian.

Share this post


Link to post
Share on other sites
On 6/11/2018 at 10:04 AM, Ulysse said:

How can I investigate on why is the robot not connecting to the server ?

I gained some experience with making Sawyer + NTP work in corporate environment. Is your problem still relevant?

In my case, I used tcpdump (Wireshark) to capture NTP traffic. It helped me to realize that Sawyer was silently rejecting new NTP configurations in situations when these configurations had accessibility limitations. The limitations of the new NTP configurations were not known to me and no indications were provided by Sawyer that it didn’t use the configured setup. I observed with Wireshark that even after updating "NTP Servers" on Sawyer, the robot continued to query NTP servers that were our OLD configuration (these old NTP servers were not available). It also became obvious that until the robot got response from new NTP servers, the setup presented to the user was not in sync with its internal configurations. A successful update happened after Sawyer was getting response from one of the NTP servers in the new setup.

From the header of your /etc/ntp.conf I see that NTP policies are managed. I have some insights about this part which may be critical to success of setting your NTP server. What company are you from? :) You can message me in private.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now