Datomic client regularly returns error on index queries

[n***@oscaro.com]

Hello,

I'm using the datomic client and a peer server with the latest version of
datomic (0.9.5561.62). Often when executing queries the first time such as
this one:

(let [db (d/db conn)
start #inst"2017-10-23"
end #inst"2017-10-24"
day-db (d/as-of (d/since db start) end)
pages (<!! (d/datoms day-db {:index :aevt
:components [:page/creation-date]
:limit 10
:chunk 10
:timeout 300000}))]
pages)

I often get a `{:cognitect.anomalies/category :cognitect.anomalies/fault,
:datomic.client/http-error nil}` error, despite setting a ridiculous
timeout. The peer logs don't say much, the peer CPU however is churning
then usually works on the 2nd and 3rd attempt. Is there some option I'm
missing perhaps?

Thanks in advance,

Nicolas

[n***@oscaro.com]

Thanks for replying :)

I indeed tried executing the same query from a peer and it was horrendously
slow but managed to complete. I have since decided that using the :aevt
index is not really practical, or seems to contain too much data to fit
into memory. I solved the problem by applying an index to the
:page/creation-date attribute and changing my code a little:

(async/thread
(.setName (Thread/currentThread) "Datom retriever")
(loop [offset 0]
(log/info "Current datom offset:" offset)
(let [chunks (d/index-range db {:attrid :page/creation-date
:start start-t
:end end-t
:offset offset
:limit limit
:chunk limit})]
(when-some [datoms (<!! chunks)]
(if (d/error? datoms)
(do
(log/error "Unable to fetch the datom stream:" datoms)
(.sleep (TimeUnit/SECONDS) 10)
(recur offset))
(let [size (count datoms)]
(do
(log/infof "Fetched datoms %s-%s" offset (+ offset size))
(doseq [datom datoms] (>!! datom-ch datom))
(when (pos? size)
(recur (+ offset size)))))))))
(log/info "Done processing!")
(async/close! datom-ch))

Works much better and my peer server is no longer at 100% CPU for 10
minutes!! What bothered me the most though is not that is was slow is that
the error message is quite unsatisfying. FYI I'm running Datomic with
DynamoDB and wasn't getting any provisioned capacity exceptions or other
errors.

Thanks anyways,

Nicolas

Hi Nicholas,
You say "the first time" -- what happens in subsequent times? Have you
tried the same query from a peer?
Thanks,
Stu

Post by n***@oscaro.com
Hello,
I'm using the datomic client and a peer server with the latest version of
datomic (0.9.5561.62). Often when executing queries the first time such as
(let [db (d/db conn)
start #inst"2017-10-23"
end #inst"2017-10-24"
day-db (d/as-of (d/since db start) end)
pages (<!! (d/datoms day-db {:index :aevt
:components [:page/creation-date]
:limit 10
:chunk 10
:timeout 300000}))]
pages)
I often get a `{:cognitect.anomalies/category :cognitect.anomalies/fault,
:datomic.client/http-error nil}` error, despite setting a ridiculous
timeout. The peer logs don't say much, the peer CPU however is churning
then usually works on the 2nd and 3rd attempt. Is there some option I'm
missing perhaps?
Thanks in advance,
Nicolas
--
You received this message because you are subscribed to the Google Groups
"Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[n***@oscaro.com]

Hi,

Originally I was running as a peer and did a query, but the number of pages
was too big (it's a web scraper) and since queries didn't support
pagination/sorting, I resorted to index scrolling like a db cursor. However
the process was still too slow and I couldn't allocate too much memory (it
is a job in a scheduler which is shared) and therefore I've been working a
lot with the peer server in the cloud. Now perhaps I can switch back to
normal queries since I can specify the offsets and limits but well this
method seems a bit more lean, and FYI there are about 40000+ page datoms
(ie. new pages) which I'm gathering per day.

Best regards

Nicolas

Hi Nicolas,
Agreed that error message is no help. The next client release will include
better error messages for a bunch of common problems.
Stepping back a bit, is there a reason you are using the raw datoms API
instead of a query? How many :page/creation-dates are there?
Stu

Post by n***@oscaro.com
Thanks for replying :)
I indeed tried executing the same query from a peer and it was
horrendously slow but managed to complete. I have since decided that using
the :aevt index is not really practical, or seems to contain too much data
to fit into memory. I solved the problem by applying an index to the
(async/thread
(.setName (Thread/currentThread) "Datom retriever")
(loop [offset 0]
(log/info "Current datom offset:" offset)
(let [chunks (d/index-range db {:attrid :page/creation-date
:start start-t
:end end-t
:offset offset
:limit limit
:chunk limit})]
(when-some [datoms (<!! chunks)]
(if (d/error? datoms)
(do
(log/error "Unable to fetch the datom stream:" datoms)
(.sleep (TimeUnit/SECONDS) 10)
(recur offset))
(let [size (count datoms)]
(do
(log/infof "Fetched datoms %s-%s" offset (+ offset size))
(doseq [datom datoms] (>!! datom-ch datom))
(when (pos? size)
(recur (+ offset size)))))))))
(log/info "Done processing!")
(async/close! datom-ch))
Works much better and my peer server is no longer at 100% CPU for 10
minutes!! What bothered me the most though is not that is was slow is that
the error message is quite unsatisfying. FYI I'm running Datomic with
DynamoDB and wasn't getting any provisioned capacity exceptions or other
errors.
Thanks anyways,
Nicolas

Hi Nicholas,
You say "the first time" -- what happens in subsequent times? Have you
tried the same query from a peer?
Thanks,
Stu

Post by n***@oscaro.com
Hello,
I'm using the datomic client and a peer server with the latest version
of datomic (0.9.5561.62). Often when executing queries the first time such
(let [db (d/db conn)
start #inst"2017-10-23"
end #inst"2017-10-24"
day-db (d/as-of (d/since db start) end)
pages (<!! (d/datoms day-db {:index :aevt
:components [:page/creation-date]
:limit 10
:chunk 10
:timeout 300000}))]
pages)
I often get a `{:cognitect.anomalies/category
:cognitect.anomalies/fault, :datomic.client/http-error nil}` error, despite
setting a ridiculous timeout. The peer logs don't say much, the peer CPU
however is churning then usually works on the 2nd and 3rd attempt. Is there
some option I'm missing perhaps?
Thanks in advance,
Nicolas
--
You received this message because you are subscribed to the Google
Groups "Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups
"Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.