Multiple d/with in a row

[Gilles Philippart]

Can I use datomic.core/with twice in a row (simulation over a simulation) ?
I do a first "update" in a sim, works ok. I update in a second sim from the
first sim db and it still shows the first sim's value.


https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172

After line 172 I expect the deal's amount to be 190,000,000 but it's still
at 180,000,000.
Am I doing it wrong ?

Gilles

[Gilles Philippart]

Thanks Stu for your answer. Ok, I understand better now.

First, I know why I get 180,000,000 from the first d/with(ed) db and don’t get 190,000,000 from the second one.
It’s because at line https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L130 <https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L130>, adding 10 ms to the db base time is long enough to have the d/with (two expressions further down the script) have the datoms inserted within that time limit and so it doesn’t get filtered by the as-of.

My intention was to demontrate how you could easily build an "alternate reality" with Datomic. One useful question for us would be : "What is we had raised this deal’s value at this particular point in time and considering all the events which we know occured aftewards, what would be our results today ? ».

What is the best way to do this in Datomic ? Is it possible to create a « detached » db from the as-of Db so that it’s not filtered (copy all the atoms up to some point) ?

Gilles

Hi Gilles,
Chaining d/with works fine, but note that d/as-of is a filter. Once you filter a database with d/as-of, it will not be able to see data added via d/with because that data is filtered from the result.
Cheers,
Stu
Can I use datomic.core/with twice in a row (simulation over a simulation) ?
I do a first "update" in a sim, works ok. I update in a second sim from the first sim db and it still shows the first sim's value.
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172 <https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172>
After line 172 I expect the deal's amount to be 190,000,000 but it's still at 180,000,000.
Am I doing it wrong ?
Gilles
--
You received this message because you are subscribed to the Google Groups "Datomic" group.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to a topic in the Google Groups "Datomic" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe <https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

[Francis Avila]

This is not general-purpose because it does not account for transactions
which manipulate schema or partitions, but it will handle the case where
you want to undo every normal data transaction back to a certain tx without
interruptions (i.e. no merge issues to consider):

http://stackoverflow.com/questions/25389807/how-do-i-undo-a-transaction-in-datomic

Hi Gilles,
You will need to use d/with to do compensating transactions to take you
back to the state you want to "branch" from.
A general purpose transaction data undoer would be a cool little sample...
:-)
Stu

Post by Gilles Philippart
Thanks Stu for your answer. Ok, I understand better now.
First, I know why I get 180,000,000 from the first d/with(ed) db and
don’t get 190,000,000 from the second one.
It’s because at line
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L130,
adding 10 ms to the db base time is long enough to have the d/with (two
expressions further down the script) have the datoms inserted within that
time limit and so it doesn’t get filtered by the as-of.
My intention was to demontrate how you could easily build an "alternate
reality" with Datomic. One useful question for us would be : "What is we
had raised this deal’s value at this particular point in time and
considering all the events which we know occured aftewards, what would be
our results today ? ».
What is the best way to do this in Datomic ? Is it possible to create a
« detached » db from the as-of Db so that it’s not filtered (copy all the
atoms up to some point) ?
Gilles
Hi Gilles,
Chaining d/with works fine, but note that d/as-of is a filter. Once you
filter a database with d/as-of, it will not be able to see data added via
d/with because that data is filtered from the result.
Cheers,
Stu

Post by Gilles Philippart
Can I use datomic.core/with twice in a row (simulation over a simulation) ?
I do a first "update" in a sim, works ok. I update in a second sim from
the first sim db and it still shows the first sim's value.
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172
After line 172 I expect the deal's amount to be 190,000,000 but it's
still at 180,000,000.
Am I doing it wrong ?
Gilles
--
You received this message because you are subscribed to the Google
Groups "Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "Datomic" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[David Greenberg]

Another thing to consider when using d/with is that it will consume memory,
so you can't create alternate realities of unbounded size (and it may have
less good performance). A more complex but more scalable way would be to
use backup/restore. You can then create new transactors on-the-fly (perhaps
by using Mesos, Chef, or Ansible) and restore the db to that transactor.
Then, you'd be able to achieve alternate realities of unbounded size.

Post by Gilles Philippart
Thanks Stu for your answer. Ok, I understand better now.
First, I know why I get 180,000,000 from the first d/with(ed) db and don’t
get 190,000,000 from the second one.
It’s because at line
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L130,
adding 10 ms to the db base time is long enough to have the d/with (two
expressions further down the script) have the datoms inserted within that
time limit and so it doesn’t get filtered by the as-of.
My intention was to demontrate how you could easily build an "alternate
reality" with Datomic. One useful question for us would be : "What is we
had raised this deal’s value at this particular point in time and
considering all the events which we know occured aftewards, what would be
our results today ? ».
What is the best way to do this in Datomic ? Is it possible to create a
« detached » db from the as-of Db so that it’s not filtered (copy all the
atoms up to some point) ?
Gilles
Hi Gilles,
Chaining d/with works fine, but note that d/as-of is a filter. Once you
filter a database with d/as-of, it will not be able to see data added via
d/with because that data is filtered from the result.
Cheers,
Stu

Post by Gilles Philippart
Can I use datomic.core/with twice in a row (simulation over a simulation) ?
I do a first "update" in a sim, works ok. I update in a second sim from
the first sim db and it still shows the first sim's value.
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172
After line 172 I expect the deal's amount to be 190,000,000 but it's
still at 180,000,000.
Am I doing it wrong ?
Gilles
--
You received this message because you are subscribed to the Google Groups
"Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "Datomic" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

[Val Waeselynck]

Hi everyone,

Regarding the ID of branching from a point in the past, what about using a
custom filter, one that will not filter out novelty added after the fact?

Here's a proof of concept:

(defn branchable-as-of [db t]
(let [t0 (d/as-of-t (d/as-of db t))
t1 (d/basis-t db)]
(d/filter db (fn [_,^Datom datom]
(not (<= t0 (d/tx->t (.tx datom)) t1))))
))


I tested it rapidly, seems to honor the expected behaviour of both with()
and asOf().

Post by David Greenberg
Another thing to consider when using d/with is that it will consume
memory, so you can't create alternate realities of unbounded size (and it
may have less good performance). A more complex but more scalable way would
be to use backup/restore. You can then create new transactors on-the-fly
(perhaps by using Mesos, Chef, or Ansible) and restore the db to that
transactor. Then, you'd be able to achieve alternate realities of unbounded
size.

Post by Gilles Philippart
Thanks Stu for your answer. Ok, I understand better now.
First, I know why I get 180,000,000 from the first d/with(ed) db and
don’t get 190,000,000 from the second one.
It’s because at line
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L130,
adding 10 ms to the db base time is long enough to have the d/with (two
expressions further down the script) have the datoms inserted within that
time limit and so it doesn’t get filtered by the as-of.
My intention was to demontrate how you could easily build an "alternate
reality" with Datomic. One useful question for us would be : "What is we
had raised this deal’s value at this particular point in time and
considering all the events which we know occured aftewards, what would be
our results today ? ».
What is the best way to do this in Datomic ? Is it possible to create a
« detached » db from the as-of Db so that it’s not filtered (copy all the
atoms up to some point) ?
Gilles
Hi Gilles,
Chaining d/with works fine, but note that d/as-of is a filter. Once you
filter a database with d/as-of, it will not be able to see data added via
d/with because that data is filtered from the result.
Cheers,
Stu

Post by Gilles Philippart
Can I use datomic.core/with twice in a row (simulation over a simulation) ?
I do a first "update" in a sim, works ok. I update in a second sim from
the first sim db and it still shows the first sim's value.
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172
After line 172 I expect the deal's amount to be 190,000,000 but it's
still at 180,000,000.
Am I doing it wrong ?
Gilles
--
You received this message because you are subscribed to the Google
Groups "Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "Datomic" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

[Val Waeselynck]

Hi everyone,

Regarding the idea of branching from a point in the past, what about using
a custom filter, one that will not filter out novelty added after the fact?

Here's a proof of concept:

(defn branchable-as-of
"Given a database value and a t (as can be passed to datomic.api/as-of),
Returns a database which only retain those facts of db that were added before t (inclusive),
but will accept new facts (as added via datomic.api/with)."
[db t]
(let [t0 (d/next-t (d/as-of-t (d/as-of db t)))
t1 (d/basis-t db)]
(d/filter db (fn [_,^Datom datom]
(not (<= t0 (d/tx->t (.tx datom)) t1))))
))


I tested it rapidly, seems to do what we expect:

(def db (current-db))
=> #'bs.utils.datomic/db

(d/q '[:find (count ?customer) . :where [?customer :customer/email]] db)
=> 39410


(def old-db (branchable-as-of db #inst "2016"))
=> #'bs.utils.datomic/old-db

(d/q '[:find (count ?customer) . :where [?customer :customer/email]] old-db)
=> 15036


(def old-db+1
(:db-after (d/with old-db
[[:db/add (d/tempid :db.part/user -1) :customer/email "***@gmail.com"]])))
=> #'bs.utils.datomic/old-db+1
(d/q '[:find (count ?customer) . :where [?customer :customer/email]] old-db+1)
=> 15037


Cheers,

Val

Hi Gilles,
You will need to use d/with to do compensating transactions to take you
back to the state you want to "branch" from.
A general purpose transaction data undoer would be a cool little sample...
:-)
Stu

Post by Gilles Philippart
Thanks Stu for your answer. Ok, I understand better now.
First, I know why I get 180,000,000 from the first d/with(ed) db and
don’t get 190,000,000 from the second one.
It’s because at line
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L130,
adding 10 ms to the db base time is long enough to have the d/with (two
expressions further down the script) have the datoms inserted within that
time limit and so it doesn’t get filtered by the as-of.
My intention was to demontrate how you could easily build an "alternate
reality" with Datomic. One useful question for us would be : "What is we
had raised this deal’s value at this particular point in time and
considering all the events which we know occured aftewards, what would be
our results today ? ».
What is the best way to do this in Datomic ? Is it possible to create a
« detached » db from the as-of Db so that it’s not filtered (copy all the
atoms up to some point) ?
Gilles
Hi Gilles,
Chaining d/with works fine, but note that d/as-of is a filter. Once you
filter a database with d/as-of, it will not be able to see data added via
d/with because that data is filtered from the result.
Cheers,
Stu

Post by Gilles Philippart
Can I use datomic.core/with twice in a row (simulation over a simulation) ?
I do a first "update" in a sim, works ok. I update in a second sim from
the first sim db and it still shows the first sim's value.
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172
After line 172 I expect the deal's amount to be 190,000,000 but it's
still at 180,000,000.
Am I doing it wrong ?
Gilles
--
You received this message because you are subscribed to the Google
Groups "Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "Datomic" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Val Waeselynck]

I withdraw my previous suggestion, which fails in subtle ways because
db.with() runs transaction functions based on the unfiltered db (tested on
datomic pro 0.9.5385).

For this reason, it seems to that db.with() is not very compatible with
filtered dbs.

Is this behavior intentional ?

Post by Val Waeselynck
Hi everyone,
Regarding the idea of branching from a point in the past, what about
using a custom filter, one that will not filter out novelty added after the
fact?
(defn branchable-as-of
"Given a database value and a t (as can be passed to datomic.api/as-of),
Returns a database which only retain those facts of db that were added before t (inclusive),
but will accept new facts (as added via datomic.api/with)."
[db t]
(let [t0 (d/next-t (d/as-of-t (d/as-of db t)))
t1 (d/basis-t db)]
(d/filter db (fn [_,^Datom datom]
(not (<= t0 (d/tx->t (.tx datom)) t1))))
))
(def db (current-db))
=> #'bs.utils.datomic/db
(d/q '[:find (count ?customer) . :where [?customer :customer/email]] db)
=> 39410
(def old-db (branchable-as-of db #inst "2016"))
=> #'bs.utils.datomic/old-db
(d/q '[:find (count ?customer) . :where [?customer :customer/email]] old-db)
=> 15036
(def old-db+1
(:db-after (d/with old-db
=> #'bs.utils.datomic/old-db+1
(d/q '[:find (count ?customer) . :where [?customer :customer/email]] old-db+1)
=> 15037
Cheers,
Val

Hi Gilles,
You will need to use d/with to do compensating transactions to take you
back to the state you want to "branch" from.
A general purpose transaction data undoer would be a cool little
sample... :-)
Stu

Post by Gilles Philippart
Thanks Stu for your answer. Ok, I understand better now.
First, I know why I get 180,000,000 from the first d/with(ed) db and
don’t get 190,000,000 from the second one.
It’s because at line
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L130,
adding 10 ms to the db base time is long enough to have the d/with (two
expressions further down the script) have the datoms inserted within that
time limit and so it doesn’t get filtered by the as-of.
My intention was to demontrate how you could easily build an "alternate
reality" with Datomic. One useful question for us would be : "What is we
had raised this deal’s value at this particular point in time and
considering all the events which we know occured aftewards, what would be
our results today ? ».
What is the best way to do this in Datomic ? Is it possible to create a
« detached » db from the as-of Db so that it’s not filtered (copy all the
atoms up to some point) ?
Gilles
Hi Gilles,
Chaining d/with works fine, but note that d/as-of is a filter. Once you
filter a database with d/as-of, it will not be able to see data added via
d/with because that data is filtered from the result.
Cheers,
Stu
On Mon, Mar 2, 2015 at 12:31 PM, Gilles Philippart <

Post by Gilles Philippart
Can I use datomic.core/with twice in a row (simulation over a simulation) ?
I do a first "update" in a sim, works ok. I update in a second sim from
the first sim db and it still shows the first sim's value.
https://github.com/gphilipp/datomic-demo/blob/master/src/datomic_demo/core.clj#L151-L172
After line 172 I expect the deal's amount to be 190,000,000 but it's
still at 180,000,000.
Am I doing it wrong ?
Gilles
--
You received this message because you are subscribed to the Google
Groups "Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "Datomic" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Datomic" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

[Marshall Thompson]

Hi Val,

Use of db.with() is incompatible with filtered values of a database.

db.with() replicates the behavior of issuing transactions (
http://docs.datomic.com/clojure/index.html#datomic.api/with) and transact
always applies to the most recent unfiltered basis-t of the database. We
will investigate throwing an exception when db.with() is called on a
filtered database value.

Best,
Marshall

[Val Waeselynck]

Hi Marshall, I would like to discuss this point further, let's take this to
a new thread.

Post by Marshall Thompson
db.with() replicates the behavior of issuing transactions (
http://docs.datomic.com/clojure/index.html#datomic.api/with) and transact
always applies to the most recent unfiltered basis-t of the database. We
will investigate throwing an exception when db.with() is called on a
filtered database value.
Best,
Marshall
--
You received this message because you are subscribed to a topic in the
Google Groups "Datomic" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/datomic/-gDB2EX7r58/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

[Dustin Getz]

A general purpose transaction data undoer would be a cool little

sample... :-)

Stu - Like this? Loading Image... :D

<http://i.imgur.com/hRXI1p7.gif>