Sorry I forgot to add that this is on Win11, python 3.11.9 with the following packages:
Package Version
------------------------- -----------
altair 5.5.0
anyio 4.8.0
attrs 25.1.0
click 8.1.8
colorama 0.4.6
docutils 0.21.2
duckdb 1.2.0
greenlet 3.1.1
h11 0.14.0
idna 3.10
itsdangerous 2.2.0
jedi 0.19.2
jinja2 3.1.5
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
loguru 0.7.3
marimo 0.11.2
markdown 3.7
markupsafe 3.0.2
narwhals 1.26.0
numpy 2.2.2
packaging 24.2
pandas 2.2.3
parso 0.8.4
polars 1.21.0
psutil 6.1.1
pyarrow 19.0.0
pycrdt 0.11.1
pygments 2.19.1
pymdown-extensions 10.14.3
pyodbc 5.2.0
python-dateutil 2.9.0.post0
pytz 2025.1
pyyaml 6.0.2
referencing 0.36.2
rpds-py 0.22.3
ruff 0.9.6
setuptools 75.8.0
six 1.17.0
sniffio 1.3.1
sqlalchemy 2.0.38
sqlglot 26.6.0
sqlrun 0.1
starlette 0.45.3
toml 0.10.2
tomlkit 0.13.2
typing-extensions 4.12.2
tzdata 2025.1
uvicorn 0.34.0
websockets 14.2
win32-setctime 1.2.0
@Josh Sarver does your server contain a large schema? we try to intropsect this schema to display it on the sidebar, but its possible that this is quite a heavy operation (and like optimized for mssql)
@Shahmir is optimizing this right now actually and making it lazy (i was the original person to make it slow), assuming that is the issue
@Josh Sarver out of curiosity, if you make the variable _eng private, does that speed it up (it should skip our auto-discovery)
Hi @Myles Scolnick , thanks so much for the response. I'm not sure what would be considered large but definitley close to a 100 tables or so. I will try to set the eng private and see what happens.
@Myles Scolnick I think you nailed it, i set the variable to _eng and it ran in 4 seconds. Honestly it did not even occur to me it was building the schema but that makes total sense now. I am suuper new to these notebooks so honestly I though this delay was occurring everytime i was querying but it definitely is not doing that so that makes teh delay much more bearable knowing it's a 1 time thing. Is it possible for it to just cache the schema or is that part fo the work you were speaking to earlier?
ah, sorry about the slowness then. its from a new feature which i did not stress tests enough.
we can likely cache the schemas in the med-term. in the short term we will make it much lazier.
is it still slow at 4 seconds? or is the expected (either from connection time or running a query)
@Myles Scolnick No I actually did a real query so I would blame that query instead, this is so much faster and honestly I should have done more digging the fact that it's only that initial schema load that is the cause makes it much less painful. I am really blown away by everything so far the embedded sql stuff is going to make life so much easier. Thanks again for the help!
glad you are enjoing it! keep the feedback coming
we don't expect you to need to _ the engine name. because by doing so, it won't show up in the dropdown for SQL cells, so we do want to fix this issue (so you can use SQL cells as intended)
Totally understand, well if there is additional questions or help/info I can provide to assist please let me know!
@Josh Sarver marimo just shipped an updated datasources sidebar (thanks to @Shahmir!) with lazier loading logic and nested sections. let me know if you have feedback after you upgrade and give it a try
Oh wow, it is a night and day improvement. I would say it took about 4 to 5 seconds to load two of those database schemas in one of my notebooks once I changed the variable to not be private anymore which is a pretty reasonable load time. Browsing the schema in the datasource pane is really fast as well. Thanks for letting me know and thanks for the fix @Shahmir! Great work guys!
@Josh Sarver, out of curiosity, do you have just the two schemas? how many tables in total?
@Myles Scolnick It was two schemas and a csv file.. I you want a more targetted test II can see how long just the one db takes.. Let me get the table count for you
@Myles Scolnick ok it was 317 tables per db, so 634 total
Ok good to know, appreciate the info
Hey guys, I'm facing a similar issue where the connection cell runs for a really long time. In my case it's a snowflake database with a huge schema, can i turn off the introspection? I'd prefer not to change the variable name and not making it private. It seems like if I make it private I can't use it in the SQL cell blocks?
Mine runs for about 2 min each time I restart the notebook
_eng = create_engine(
URL(
user="xxxx",
account="xxx",
warehouse="xxx",
database="xxx",
authenticator="externalbrowser",
)
)
pd.read_sql("SELECT 1;", _eng)runs for 5 sec
engine = create_engine(
URL(
user="xxxx",
account="xxx",
warehouse="xxx",
database="xxx",
authenticator="externalbrowser",
)
)
pd.read_sql("SELECT 1;", engine)runs for 2 min
Hey, sorry about that. Yeah, I'm thinking if it should be opt-in instead for now.
In case you know, is the introspection working correctly / showing error logs? Wanted to check if introspection is failing hence the issue
it seems to be working correctly, I can see all schema's and tables under the data sources in the helper panel. Which is really great but it's slowing me down too much unfortunately given the size of the information_schema and how often I restart the kernel while testing things
Hey @RyanFras, we made introspection opt-in (disabled it by default) in the latest release. Should help with the issue that you're facing
docs to opt-inawesome, I tested it and works great now. super fast. thanks so much for the quick help!
