A recent incident which occurred to me and a colleague of mine made me write this little post. We were setting a property in a hive script and was trying to retrieve some data comparing with the value we set. In other words, this is what we did.
Unix timestamp gives the number of seconds which has passed since the epoch.
Java timestamp gives the number of milliseconds since the epoch.
Although both of these uses the same epoch (midnight of January 1st 1970), they provide two different values. Thats why the query was not behaving as intended. Although I had read/learned about this difference, it took some time to kick it into the mind.
- We set a property named max_ts to hive context within a Java class. We used the Timestamp.getTime() method to get the time.
- In the hive script, we were doing something like:
- Select my_columns from my_table where unix_timestamp(a_string) > ${hiveconf:max_ts}
Unix timestamp gives the number of seconds which has passed since the epoch.
Java timestamp gives the number of milliseconds since the epoch.
Although both of these uses the same epoch (midnight of January 1st 1970), they provide two different values. Thats why the query was not behaving as intended. Although I had read/learned about this difference, it took some time to kick it into the mind.