[Gravity] added support for recursive common table expressions in gravity

Review Request #544 — Created Feb. 15, 2018 and submitted

k.hengsdijk
Lunr
recursiveCTEsupport
lunr

added support for recursive common table expressions to gravity



  • 0
  • 0
  • 32
  • 3
  • 35
Description From Last Updated
pprkut
  1. Please rebase on top of https://reviews.lunr.nl/r/542/
    Too much duplicated code

  2. 
      
k.hengsdijk
pprkut
  1. This is still not incremental. It still shows everything as new. Please make sure you reference your parent properly

  2. 
      
k.hengsdijk
pprkut
  1. 
      
  2. src/Lunr/Gravity/Database/DMLQueryBuilderInterface.php (Diff revision 3)
     
     
     
     
     
     
     
     
     
     

    While this certainly works, it shoves all the responsibility to create the right query somewhere else, either to the developer or to the escaper. And I can't really think of a escaper function name that wouldn't be confusing or hard to identify. recursive_with_query is a mouthful...

    We can do better ;)

    Let's see if you can come up with a better solution than this on your own ;)

  3. 
      
k.hengsdijk
smillernl
  1. 
      
  2. Still needs a description

  3. Descriptions should start with a capital letter

  4. doesn't it contain the whole WITH query?

  5. Do CTEs only work for SELECT queries?

    1. yeah they only work with select queries because a CTE creates a virtual table kinda like a view in SQL.

  6. Same as before

  7. there should probably be a space between the alias and the collumns

  8. 
      
k.hengsdijk
pprkut
  1. 
      
  2. src/Lunr/Gravity/Database/DMLQueryBuilderInterface.php (Diff revision 5)
     
     
     
     
     
     
     
     
     
     
     
     

    This is really clumsy for various reasons:

    1) can't use recursive CTEs without specifying column_names. If you reverse the order you can't specify column_names without declaring a recursive part. Ideally you want neither.

    2) Parameter names are misleading. $recursive is part of the SQL query, so what is the extra $sql_query about?

  3. what about UNION ALL?

  4. 
      
k.hengsdijk
k.hengsdijk
pprkut
  1. 
      
  2. src/Lunr/Gravity/Database/DMLQueryBuilderInterface.php (Diff revision 7)
     
     
     
     
     
     
     
     
     
     
     

    This is a good step in the right direction, but you didn't have to throw away all the other code :/

    $sql_query alone is too lazy. We can offer an interface for the developer that better covers the WITH RECURSIVE use case.

  3. 
      
k.hengsdijk
pprkut
  1. 
      
  2. Sorry, that's wishful thinking.

    Individual query parts are meant to be able to be constructed out-of-order. So conceptually, I can define a union upfront, and construct from/where/having/etc later on based on other conditions.

    But with this code, I could no longer do that, since the upfront defined union would conflict with a recursive CTE and the query constructed would not at all be what I want it to look like.

    1. ah i thought it would work because i empty the union right after calling it but i didnt think about what if someone wants to create a union before the call. so if i adapt this solution with a different way to construct the union between the two select queries would it work then?

  3. 
      
k.hengsdijk
pprkut
  1. 
      
  2. When reading this, nobody has any clue wtf this is about. Why would I ever not want all results?

  3. with_recursive()

  4. src/Lunr/Gravity/Database/SQLDMLQueryBuilder.php (Diff revision 9)
     
     
     
     
     
     

    query constructing should happen in sql_with()

    Don't make the larger use-case fit the smaller interface.

  5. 
      
k.hengsdijk
smillernl
  1. 
      
  2. This is confusing, you'd think union = true would mean that it's using UNION

  3. src/Lunr/Gravity/Database/DatabaseDMLQueryBuilder.php (Diff revision 10)
     
     
     
     
     

    consistent uppercasing for types please

  4. $this or $self?

  5. 
      
k.hengsdijk
smillernl
  1. Missing tests

  2. 
      
k.hengsdijk
smillernl
  1. 
      
  2. why make this uppercase?

  3. strict comparisons

    1. this is also the done the same way in the rest of the class so i also used it in order to be consistent.

  4. src/Lunr/Gravity/Database/SQLDMLQueryBuilder.php (Diff revision 12)
     
     
     
     
     

    Please don't use uppercase for classes.

  5. 
      
k.hengsdijk
pprkut
  1. 
      
  2. I would not bind this to the alias. Either bind it to the keyword (WITH) or make it a separate variable

  3. src/Lunr/Gravity/Database/DatabaseDMLQueryBuilder.php (Diff revision 13)
     
     
     
     
     
     
     
     
     
     

    recursive statements only work in the else clause?

    1. yes the recursive part has to be the the first part of the sql query otherwise it will not work.

    2. but can there be more than one? I don't see a restriction in the grammar, nor a rule that one has to come before the other. Can you point me to where you found that?

    3. to my knowledge there can only be one recursive with statement in a sql query and it has to be the first with statement so thats why it only happens there in the else statement. i have tried to construct querys with the recursive part in other places and tried making a query with two recursives with statements but they always failed. Also i could not find a example of this anywhere on the internet.

    4. I understand that this might seem odd, but I still don't see that restriction in the grammar (for sqlite for example: https://www.sqlite.org/lang_with.html)
      The goal is to support the functionality from the database, not prevent the user from shooting himself in the foot ;)

    5. yeaahh i see what you mean i did do it to prevent the user from shooting himself in the foot. mariadb has same grammar as that example in the link so i will change it.

  4. 
      
k.hengsdijk
k.hengsdijk
pprkut
  1. 
      
  2. src/Lunr/Gravity/Database/DatabaseDMLQueryBuilder.php (Diff revision 15)
     
     
     
     
     

    I don't think that's correct though. If you check the grammar, the "RECURSIVE" keyword is only mentioned once in the beginning. You can't have that one more than once.
    What I read is that it needs to be present as soon as one of the later mentioned CTEs is recursive.
    However, since we can specify CTEs in any order in the code, we don't know with the first statement whether there's a recursive one coming later. That's a fun one to solve :)

  3. 
      
k.hengsdijk
pprkut
  1. 
      
  2. what if $temp already includes a recursive CTE?

    It really helps adding unit tests for the cases that you want to implement. Think about all the cases that could happen, add tests for them, and then make them work.

  3. 
      
k.hengsdijk
pprkut
  1. 
      
  2. src/Lunr/Gravity/Database/Tests/DatabaseDMLQueryBuilderQueryPartsWithTest.php (Diff revisions 16 - 17)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

    what about testNonRecursiveWithAfterRecursiveQuery*?

  3. 
      
k.hengsdijk
k.hengsdijk
pprkut
  1. 
      
  2. This doesn't work, as the semantic meaning of what's written here means "put WITH statements between WHERE and GROUP BY".

    I forgot that SELECT isn't part of the implode, so you'll likely have to handle the "WITH (RECURSIVE)" part directly here (in get_select_query()).

  3. 
      
k.hengsdijk
pprkut
  1. 
      
  2. 'recursive' is not enough as a variable name. Missing context

  3. src/Lunr/Gravity/Database/DatabaseDMLQueryBuilder.php (Diff revision 20)
     
     
     
     
     
     
     
     
     

    You can't write into $this->with. get_*() methods must be without side-effects (calling it twice should yield the same result)

  4. I would use NULL as default value everywhere, but still handle empty strings as well

  5. You don't need a temporary variable.

    1. but i need to empty the $this->with string and later add it to the end of the new with string. I dont see any other way of doing that except with a $temp variable

    2. You don't need to empty $this->with. And puff you're need for $temp is gone ;)

    3. but if i dont empty $this->with it is impossible for the user to specify a recursive query after a normal query in the code. A recursive query always has to be the first part of the query so if $this->with is not emptied i would have to use some string manipulation the add the recursive query to the front of the query which would be expensive.

    4. Why? You construct a new string. Whether you have the new one first and the old one second or the old one first and the new one second doesn't matter.

  6. what if UNION is not set?

  7. 
      
k.hengsdijk
k.hengsdijk
pprkut
  1. 
      
  2. src/Lunr/Gravity/Database/DatabaseDMLQueryBuilder.php (Diff revisions 21 - 22)
     
     
     
     
     
     
     

    Much better :)

    Now merge this if with the if below and we are done ;)

  3. 
      
k.hengsdijk
pprkut
  1. Ship It!
  2. 
      
k.hengsdijk
Review request changed

Status: Closed (submitted)

Change Summary:

Merged into master

Loading...