rdd.countApprox taking as long as count()

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP

rdd.countApprox taking as long as count()



My code looks like


foo.rdd.countApprox(1000, 0.9) => takes 7.1 minutes
foo.count() => takes 7.1 minutes



Is there anything I am missing? foo is a df and I am trying to reduce the time it takes to count() the number of records in foo


foo


count()


foo



As you can see below it some cases it takes more time and in some other cases, it takes lesser time. I am confused :(



enter image description here



enter image description here





Seems like you should be comparing the first to foo.rdd.count()
– pault
8 mins ago


foo.rdd.count()









By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Comments

Post a Comment

Popular posts from this blog

Executable numpy error

Image
Clash Royale CLAN TAG #URR8PPP Executable numpy error I'm trying to create an executable file for my code. I've already tried with cx_Freeze and with pyinstaller. Both gives me the same error, which is: "Missing required dependencies 0.format(missing_dependencies))" PS C:UsersGustavoDesktopbuildexe.win32-3.6> python AgendaOficial.py C:UsersGustavoAppDataLocalProgramsPythonPython36-32python.exe: can't open file 'AgendaOficial.py': [Errno 2] No such file or directory PS C:UsersGustavoDesktopbuildexe.win32-3.6> .AgendaOficial.exe Traceback (most recent call last): File "C:UsersGustavoAppDataLocalProgramsPythonPython36-32libsite-packagescx_Freezeinitscripts__startup__.py", line 14, in run module.run() File "C:UsersGustavoAppDataLocalProgramsPythonPython36-32libsite-packagescx_FreezeinitscriptsConsole.py", line 26, in run exec(code, m. dict ) File "AgendaOficial.py", line 8, in File "C:UsersGustavoAppDataLocal
Read more

PySpark count values by condition

Image
Clash Royale CLAN TAG #URR8PPP PySpark count values by condition I have a DataFrame, a snippet here: [['u1', 1], ['u2', 0]] basically one string ('f') and either a 1 or a 0 for second element ('is_fav'). What I need to do is grouping on the first field and counting the occurrences of 1s and 0s. I was hoping to do something like num_fav = count((col("is_fav") == 1)).alias("num_fav") num_nonfav = count((col("is_fav") == 0)).alias("num_nonfav") df.groupBy("f").agg(num_fav, num_nonfav) It does not work properly, I get in both cases the same result which amounts to the count for the items in the group, so the filter (whether it is a 1 or a 0) seems to be ignored. Does this depend on how count works? count 1 Answer 1 There is no filter here. Both col("is_fav") == 1 and col("is_fav") == 0) are just boolean expressions and count doesn't really care about their value as long a
Read more

Trying to Print Gridster Items to PDF without overlapping contents

Image
Clash Royale CLAN TAG #URR8PPP Trying to Print Gridster Items to PDF without overlapping contents I want to print gridster items to pdf using chome browser. Unfortunately the item overlaps on next page. Is there any way to make gridster item go to next page if it doesnt fit in the current page when printing to pdf? Screenshot By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
Read more